Repository navigation
Commit f096d8c
fix(drivers): resolve warehouse SDKs from disk instead of reporting them missing (#1122)
* fix(drivers): resolve warehouse SDKs from disk instead of reporting them missing
A bare `import("snowflake-sdk")` inside the compiled Bun binary resolves
against bunfs, which has no `node_modules`. An SDK the user had already
installed was invisible to the runtime, which then reported it as "not
installed" — the single root cause behind nine open issues, five of which
were filed automatically by the telemetry scanner.
Add `packages/drivers/src/resolve.ts` and route all twelve drivers through
it:
- `loadOptionalDriver()` tries the ambient resolver first (unchanged
behaviour in dev and the monorepo), then resolves against real
directories: the managed install dir, `ALTIMATE_BIN_DIR`, `NODE_PATH`,
the project and its parents, and the executable's own tree.
- Installs land in `<XDG_DATA>/altimate-code/drivers`, which no upgrade
path touches. `~/.altimate/bin` is rebuilt by the curl installer's
self-upgrade, which is how hand-installed drivers were being wiped.
- A package that is present but fails to load is now reported as a broken
install rather than a missing one, so users are not sent to reinstall
what they already have.
- `DriverNotInstalledError` names the exact install command and every
location searched, replacing twelve copies of a bare `npm install` hint.
Also add the `warehouse_install_driver` tool, and have `warehouse_add`
report driver readiness at the point it can still be acted on. The check is
filesystem-only and deliberately does not install: adding a connection must
not block on a network `npm install`.
Fix pre-existing drift in the driver catalogue. `mongodb` had a driver
module and a workspace dependency but was missing from the binary's
`optionalExternals` (so it was bundled instead of installed on demand) and
from the published package's optional peer dependencies (so it was never
surfaced to users). `driver-catalogue.test.ts` now holds all four
declaration sites to `DRIVER_PACKAGES`.
Verified in the environment the bug actually occurs in: compiled a binary
with the production `Bun.build` options and confirmed bare `import("pg")`
fails with `Cannot find package 'pg' from '/$bunfs/root/…'` while
`loadOptionalDriver` loads the real module. Same for the subpath
(`mysql2/promise`) and scoped (`@clickhouse/client`) specifier shapes.
Tests: 162 drivers unit, 4,712 opencode, 140 Docker-backed driver e2e
(Postgres, DuckDB, ClickHouse, MongoDB, data-diff), 29 real-Snowflake
finops e2e. Typecheck clean.
Closes #671
Closes #295
Closes #1075
Closes #61
Closes #769
Closes #764
Closes #713
Closes #670
Closes #659
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(drivers): address consensus review — additive installs, half-installed packages, azure auth, type aliases
Findings from the multi-model consensus review of #1122. Each was reproduced
before being fixed.
**Installing a driver deleted the previous one.** `installOptionalDriver` ran
`npm install --no-save`, so npm treated every already-installed driver as
extraneous and pruned it. Reproduced on npm 11.12.1: installing `mysql2` into a
prefix holding `pg` printed `added 12 packages, and removed 14 packages`. A user
adding a second warehouse silently lost the first — re-creating the exact defect
this module exists to fix. The install now saves to the directory's manifest,
which makes it genuinely additive (verified across three drivers).
**A half-installed package reported as installed.** `resolveOptionalPackage`
fell back to returning the package directory when `require.resolve` failed, so
an empty `node_modules/pg` resolved successfully and `isDriverInstalled` was
true. `warehouse_install_driver` then answered "already installed, no action
taken" and the driver could never be repaired. Resolution now requires a
manifest and an entry file that exists, and keeps searching later roots instead
of returning a path the caller cannot import.
**Azure AD auth used the pattern this PR removes.** `sqlserver.ts` still called
`import("@azure/identity" as string)`, which cannot resolve inside the compiled
binary, so an installed `@azure/identity` was invisible and every Azure AD login
silently fell through to the az CLI. Routed through a new `loadOptionalPackage`
(soft variant that returns undefined rather than throwing, since this caller has
a real fallback), and declared as a non-driver external.
**Six warehouse types never got a readiness note.** `DRIVER_MAP` routes 18 type
strings onto 13 drivers, but `driverForWarehouseType` matched only the 12
canonical names, so a connection added as `postgresql`, `mariadb`, `mssql`,
`fabric` or `mongo` skipped the check added for #61 — the silent-broken-
connection case that issue is about.
**Test quality.** The review mutation-tested `isModuleNotFound` by deleting it
and all 22 tests still passed; its fixture was never ambiently resolvable, so
the branch was unreachable. Applying the same technique to the new fixes showed
the first half-installed test was also vacuous. `isModuleNotFound` and
`npmInstallArgs` are now exported and pinned directly, and four mutants — always-
missing predicate, `--no-save` restored, manifest check removed, bare-directory
return — each fail at least one test.
Tests: 172 drivers unit (was 162), 4,712 opencode, 140 Docker-backed driver e2e.
Typecheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(drivers): address PR bot review — transitive deps, load probe, telemetry, quoting
CodeRabbit and cubic-dev-ai findings on #1122. Each verified before fixing.
**A missing transitive dependency read as a missing driver.** `isModuleNotFound`
matched any "Cannot find module/package" text, but a driver whose own dependency
tree is incomplete raises exactly that shape — observed for real inside a
compiled binary as `Cannot find package 'pg-protocol' from '.../pg/lib/
connection.js'`, where pg itself is installed. The predicate now takes the
specifier and, when the runtime names the module it could not find, only counts
a name matching what was asked for. Without a specifier it stays conservative.
**A broken install could not be repaired.** `warehouse_install_driver` gated on
`isDriverInstalled`, which only asks whether the package resolves. A copy that
resolves but throws on import — a native addon for another platform, or a
half-written install — answered "already installed", so the one command that
could fix it declined to run. It now probes an actual load.
**Failed installs were recorded as successes.** `Tool` reads
`metadata.success === false` as its soft-failure signal (tool/tool.ts), and every
sibling warehouse tool sets it. This tool omitted it, so a failed install skipped
failure telemetry entirely.
**Install hints broke on paths containing spaces.** The printed
`npm install --prefix <dir>` is meant to be pasted; an unquoted path split and
npm received the wrong prefix. Added `shellQuote` and applied it at both sites.
**Two test-quality fixes.** CodeRabbit and cubic independently flagged that
"does not fall back when an ambiently-resolvable package fails to load" never
reaches the branch it names — its fixture is not ambiently resolvable, so the
disk fallback handles it first. Renamed to what it actually proves, with the
ambient branch now pinned directly through `isModuleNotFound`. Separately, a
comment claimed the catalogue test kept the tool's `DRIVER_NAMES` and alias map
in step with `DRIVER_PACKAGES`; no such test existed. It does now, and it also
asserts every `DRIVER_MAP` type resolves to an installable driver — removing the
alias map fails it, which is the #61 gap this PR set out to close.
Tests: 177 drivers unit (was 172), 4,714 opencode. Typecheck clean, 0 lint errors.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(drivers): make the repair path actually repair, and stop a broken copy shadowing a good one
Second bot round on #1122. The headline finding is that my previous commit's
repair path did not work.
**The reinstall never ran.** `warehouse_install_driver` gained a load probe so a
resolvable-but-unloadable driver would be rebuilt — but `installOptionalDriver`
short-circuits on `isDriverInstalled`, a resolution-only check, and returned
`installed: true, alreadyPresent: true` without invoking npm. The probe changed
nothing and the tool reported a success it had not performed. cubic-dev-ai
flagged this three times over. Installs now take a `force` option for callers
that know something the resolution check cannot, and the tool passes it exactly
when the package resolves but fails to import.
**A broken ambient copy hid a healthy managed one.** After an ambient import
failed with anything other than a resolution error, the loader rethrew
immediately, so installing a good copy into the managed directory could never
take effect. Resolution now continues to the search roots, and the ambient error
is only surfaced when nothing else loads.
**Concurrent installs could corrupt the managed directory.** Two installs
running npm against one manifest are serialized per target directory.
**Windows install hints were unusable.** `shellQuote` emitted POSIX single
quotes, which cmd.exe and PowerShell do not understand, so any path containing a
space produced a command that could not be run. It is now platform-aware.
**Test honesty.** The catalogue test asserted only that a registry type resolved
to *something*; a stale alias naming an uninstallable driver would have passed.
It now checks membership in DRIVER_PACKAGES. More importantly, the first attempt
at the ambient-shadowing test was vacuous in the same way three earlier tests
were — its fixture was not ambiently resolvable, so the branch under test was
never reached, and the mutant survived. It now writes a genuinely
ambient-resolvable fixture into this package's node_modules and removes it
afterwards. Mutants for all three fixes were confirmed to fail.
Tests: 182 drivers unit (was 177), 4,714 opencode. Typecheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(drivers): make the repair actually rebuild, and fix the install queue race
Third bot round on #1122. Two findings were raised independently by both
CodeRabbit and cubic-dev-ai, which is what made them worth checking closely.
**The repair still did not repair.** `force` skipped the resolution-only early
return, but `performInstall` then ran an ordinary `npm install`. npm compares the
manifest against what is recorded, not the health of what is on disk, so with the
package already present it answers "up to date" and rewrites nothing. Verified on
npm 11.12.1 against a deliberately corrupted `pg`: the corrupt file survived.
cubic proposed appending `--force`. That does not work either — tested, and the
corrupt copy still survived, because `--force` forces *fetching* rather than
overwriting an already-satisfied dependency. What does work is deleting the
package directory first, so a repair now does that before invoking npm.
**The install queue serialized only two callers.** Awaiting the in-flight promise
released everyone waiting on it at once, and each continuation then started its
own `performInstall` without re-reading the map. With three or more installs the
later ones overlapped on the same manifest — the exact condition the block exists
to prevent. Installs now chain onto the current tail instead.
**Two tests were not hermetic.** The forced-install test spawned a real
`npm install oracledb` against the live registry, so a unit test depended on npm
being on PATH and on network access, with a 15s timeout to block on. The ambient
tests wrote a throwing package into this package's real `node_modules`, which a
killed run would have left behind to break later resolutions.
Both now use injection: `installOptionalDriver` takes a `runNpm`, and
`loadOptionalDriver` takes an importer. That keeps the ambient-failure branch
genuinely exercised — the reason the fixture was written to disk in the first
place — without touching the dependency tree or the network. The file now runs in
~100ms with no external dependencies.
Mutants confirmed failing: repair that skips the delete, the old await-then-start
queue, and `force` ignored entirely.
Tests: 185 drivers unit (was 182), 4,565 opencode. Typecheck clean, 0 lint errors.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(drivers): kill the npm process tree on timeout, and quote the last install hint
Kilo Code's review, which finally ran after its provider rate limit cleared.
**A timed-out install kept running.** `runNpm` spawns with `shell: true`, so the
child is the shell and npm is its descendant; `child.kill()` therefore reaped the
shell and left npm writing to the shared driver directory after the promise had
already resolved with a timeout error. The in-flight map cannot help here — it
serializes promises, not processes — so a subsequent install could overlap an
orphan. POSIX runs now get their own process group (`detached`) and are killed
group-wide; Windows, which has no process groups, uses `taskkill /T /F`.
**One install hint was still unquoted.** `warehouse_install_driver`'s failure
branch interpolated the directory raw while the two equivalents in
`DriverNotInstalledError` and `driverReadinessNote` were already wrapped in
`shellQuote`. That is the branch a user reads precisely when the automatic
install has failed them, and on a path like `C:\Users\John Doe\...` the copied
command splits and npm receives the wrong prefix. A test now asserts every
printed `--prefix` is quoted.
Also merges origin/main (12 commits). The only conflict was in the driver
externals list, where main had reformatted it one-per-line while this branch
added mongodb, @clickhouse/client, trino-client and @azure/identity; the
resolution keeps main's formatting and this branch's four additions.
Tests: 186 drivers unit, 4,696 opencode. Typecheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(drivers): handle taskkill's async spawn error, and quote the last --prefix site
Fourth bot round on #1122, both findings against the previous commit.
**An unhandled `error` event could take the process down.** `killTree` wrapped
`spawn("taskkill", …)` in try/catch, but spawn reports a missing binary through
an asynchronous `error` event rather than a throw, so the catch never ran. An
unhandled `error` on a ChildProcess is fatal — meaning a Windows timeout could
kill the CLI while leaving the npm install it was trying to stop still running.
The killer now carries an `error` handler that falls back to killing the child
directly.
**A third `--prefix` site was still unquoted.** `installOptionalDriver`'s
npm-missing branch built its hint from the raw directory, so a user without npm
whose driver directory contains a space — a Windows profile, say — got a
copy-paste command that splits on it.
The test that should have caught that claimed "every printed --prefix is quoted"
while only ever exercising `DriverNotInstalledError`. It is replaced by
behavioural cases for each message-producing branch plus a structural check that
scans the sources and fails on any `--prefix ${…}` not wrapped in `shellQuote`.
Verified by adding a brand-new unquoted hint in an unrelated function: the
structural check fails, which is the failure mode that let this site through
twice.
Tests: 188 drivers unit (was 186), 4,696 opencode. Typecheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(drivers): settle a timed-out install only once the npm tree is gone
Addresses the open review findings on this PR.
`killTree` started the kill and returned. On Windows `taskkill` runs as
its own child process, so `runNpm` settled while the timed-out npm tree
was still writing to the shared driver directory and the next queued
install could overlap it — `installsInFlight` serializes promises, not
processes. The POSIX path had the same hole for a different reason:
`process.kill(-pid, SIGTERM)` only *signals*, and delivery is
asynchronous, so the group was routinely still alive when the promise
resolved. `killTree` is now awaitable and resolves when the process has
actually exited, with a SIGKILL escalation so a group that ignores
SIGTERM cannot stall the queue.
The timeout path had no test coverage at all. The new test asserts the
contract that matters — the tree is dead by the time the promise
resolves, not merely that a kill was requested — and fails against the
previous implementation.
`driverSearchRoots` gains the drivers package's own location. An SDK
hoisted next to an installed `@altimateai/drivers` resolves at require
time but was invisible to the roots list, so `isDriverInstalled`
reported a working driver as missing and the readiness note nagged.
Two test fixes: the `DriverNotInstalledError` quote assertion hardcoded
`'`, which fails on Windows where `shellQuote` emits `"`; and the
catalogue bound `toBeGreaterThan(12)` duplicated the driver count and
was trivially true, so it is replaced by the invariant it was reaching
for — every installable driver is reachable from a registry type.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018fJ3X7pcGT4R9yzjsJnqsV
* fix(drivers): make install teardown and readiness atomic
* fix(drivers): enforce optional SDK trust boundaries
* fix(drivers): reject invalid clickhouse ports
* test(drivers): cover forced repair permissions
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: anandgupta42 <[email protected]>1 parent 5993471 commit f096d8c
22 files changed
Lines changed: 2563 additions & 105 deletions
File tree
- packages
- drivers
- src
- test
- opencode
- script
- src
- altimate/tools
- tool
- test/altimate
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
| 10 | + | |
16 | 11 | | |
17 | 12 | | |
18 | 13 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
9 | 51 | | |
10 | 52 | | |
11 | 53 | | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
20 | 58 | | |
21 | 59 | | |
22 | 60 | | |
23 | 61 | | |
24 | 62 | | |
25 | 63 | | |
26 | | - | |
27 | | - | |
28 | | - | |
| 64 | + | |
29 | 65 | | |
30 | 66 | | |
31 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
| 10 | + | |
| 11 | + | |
17 | 12 | | |
18 | 13 | | |
19 | 14 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
| 10 | + | |
| 11 | + | |
15 | 12 | | |
16 | 13 | | |
17 | 14 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| |||
130 | 131 | | |
131 | 132 | | |
132 | 133 | | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
| 134 | + | |
| 135 | + | |
139 | 136 | | |
140 | 137 | | |
141 | 138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
| 10 | + | |
| 11 | + | |
15 | 12 | | |
16 | 13 | | |
17 | 14 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
| 10 | + | |
| 11 | + | |
18 | 12 | | |
19 | 13 | | |
20 | 14 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
| 10 | + | |
14 | 11 | | |
15 | 12 | | |
16 | 13 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
| 11 | + | |
17 | 12 | | |
18 | 13 | | |
19 | 14 | | |
| |||
0 commit comments