Skip to content

Commit 1caa234

Browse files
anandgupta42claude
andauthored
fix: --dir silently read a populated warehouse store as empty (#1204)
* fix: [#1203] --dir silently read a populated warehouse store as empty `--dir` calls `process.chdir()` (`cli/cmd/run.ts:412`). `ConnectionRegistry` never absolutized the `path` field, so a relative store path in a connection config resolved against whatever working directory the process happened to have when the driver opened it. Both file-backed drivers then answered the miss by CREATING an empty database — `new duckdb.Database(path)` in `drivers/src/duckdb.ts`, `create: !isReadonly` in `drivers/src/sqlite.ts`. Every query afterwards succeeded and returned zero rows, with no error and no fault signal for a caller to key on. Two fixes, either of which breaks the chain; both are needed because the second is the root hazard and is independent of `--dir`: 1. Resolve store paths deterministically. `resolveStorePaths()` absolutizes a relative `path` once, at config-load time, against the directory that declared it — global config against `~/.altimate-code`, project config and `ALTIMATE_CODE_CONN_*` against the project root. A later `chdir` cannot move it. `warehouse_add` persists the absolute form for the same reason. The project root comes from `Instance.directory`, not `process.cwd()`. A server or `run --attach` session never chdirs — it carries the project in the instance context and leaves cwd at the server's launch directory — so cwd there is somebody else's project. `localConfigPath()` had that bug already; absolutizing at load time would have baked it in permanently. 2. Never conjure a store that was meant to exist. New `drivers/src/file-store.ts` holds `assertStoreExists()`, which both file-backed drivers call before opening: a missing local file throws an error naming the path it looked for. Creation is opt-in via `create: true`, which `schema_sync` and `test_local` — the two tools that deliberately materialize a scratch store — now pass. Exempt from the check: the exact string `:memory:`, an empty path, and scheme-qualified targets (`md:`, `s3://`, `ducklake:`, anything a DuckDB extension provides), which the driver reports as an unknown scheme rather than creating. Only the EXACT `:memory:` — DuckDB writes a real file for `:memory:named` and for `:foo`, so those stay inside the guard. Windows drive letters stay paths. `bun:sqlite` rejects an options object with no open flag, so the SQLite driver now passes `readwrite` explicitly where `create` used to imply it. Relative SQLite `file:` URIs are deliberately NOT handled here, and are filed as #1209 with the evidence. They follow cwd the same way, but `bun:sqlite` parses `file:` as a URI on macOS and appears to treat it as a literal filename on Linux, and Windows is unverified. A rewrite changes which database opens, so shipping one across platforms it has not been proven on would create the same class of defect this fixes. Verified through a compiled binary built with the production build options, invoked with `--dir` from an unrelated working directory. Before: `tables: []` plus a stray `warehouse.db` (+ `-wal`, `-shm`) written into the `--dir` target; the DuckDB arm left a 12,288-byte stray `.duckdb`. After: the real table, and nothing created. `test/altimate/store-path-resolution.test.ts` compiles that binary and runs five arms — including a server-style request driven through `Instance.provide` with a decoy config and store in the launch directory — of which three fail on unmodified main. Existing tests that relied on create-on-open to build their own scratch store now pass `create: true`. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix: [#1203] a broken warehouse connection reached the agent as "(0 rows)" Field verification against the real rig showed the first version of this fix did not help: `--dir` still returned "no tables", nothing was created anywhere on disk, and `assertStoreExists()` produced no visible error even when pointed at a store that does not exist. The diagnosis was wrong, not the measurement. A failure was shaped like success at every layer between the driver and the model, so three separate things had to be true for the agent to be misled, and all three were: 1. `drivers/src/duckdb.ts` and `drivers/src/sqlite.ts` read `(config.path as string) ?? ":memory:"`. Any failure to carry a path — a config the registry never loaded, a field under another name, a lookup that fell through — became a successful connection to an empty in-memory database. That is worse than the silent creation this PR already closed: a stray file can at least be found afterwards, whereas this leaves nothing on disk to explain the empty answer. `requireStorePath()` now rejects a missing or empty path. `:memory:` is still available, but only when asked for. 2. `sql.execute` never throws. It catches every connection and query error and returns `{ columns: [], rows: [], row_count: 0, error }` (native/connections/register.ts:437). So the loud error from (1) and from `assertStoreExists()` was demoted to a field on a success-shaped result. 3. `formatResult()` in `tools/sql-execute.ts` returned the literal string `"(0 rows)"` whenever `row_count === 0` and never read `error`. An agent running `SELECT ... FROM information_schema.tables` against an unresolvable warehouse was told, in effect, that the warehouse was healthy and empty — then offered a tip about query optimization. `sql_execute` now surfaces the error the way `schema_inspect` already did, via `normalizeError`. So `assertStoreExists()` was never bypassed. It fired, and its message was thrown away one layer above. A guard whose failure cannot reach the caller is not a guard, which is why the new tests drive the layers the product actually uses: the compiled-binary arms now go through `Dispatcher.call("sql.execute")` rather than reaching the driver via `Registry.get()`, and the rendering is tested through the `sql_execute` tool itself. `test/altimate/warehouse-failure-visibility.test.ts` covers all three: a pathless config, a missing store, and an error-carrying result that must not render as `(0 rows)` — plus the two cases that must NOT change, an explicit `:memory:` and a genuinely empty result set from a healthy warehouse. Three of its six fail on unmodified main, as do three of the five compiled-binary arms. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 546781d commit 1caa234

21 files changed

Lines changed: 1050 additions & 30 deletions

‎docs/docs/configure/warehouses.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,26 @@ If you're already authenticated via `gcloud`, omit `credentials_path`:
262262
| Field | Required | Description |
263263
|-------|----------|-------------|
264264
| `path` | No | Database file path. Omit or use `":memory:"` for in-memory |
265+
| `create` | No | Create the database file if it is missing (default: `false`) |
266+
267+
!!! warning "The store must already exist"
268+
Connecting never creates the database. If the file is missing, the connection
269+
fails with an error naming the path it looked for — an empty database would
270+
otherwise answer every query with no rows and no error, which reads as a
271+
successful query against an empty warehouse. Set `"create": true` only when
272+
you intend this store to be created.
273+
274+
!!! note "How a relative `path` is resolved"
275+
A relative `path` is resolved once, when the config is loaded, against the
276+
directory that declares it:
277+
278+
- `~/.altimate-code/connections.json` → resolved against `~/.altimate-code`
279+
- `<project>/.altimate-code/connections.json` → resolved against `<project>`
280+
- `ALTIMATE_CODE_CONN_*` environment variables → resolved against the project root
281+
282+
It is never resolved against the current working directory, so `--dir` cannot
283+
re-point an existing connection at a different file. Absolute paths are always
284+
safest.
265285

266286
!!! note "Concurrent access"
267287
DuckDB does not support concurrent write access to the same file. If another process holds a write lock, Altimate Code automatically retries the connection in **read-only** mode so you can still query the data. A clear error message is shown if read-only access also fails.
@@ -443,10 +463,16 @@ If you're already authenticated via `gcloud`, omit `credentials_path`:
443463
|-------|----------|-------------|
444464
| `path` | No | Database file path. Omit or use `":memory:"` for in-memory |
445465
| `readonly` | No | Open in read-only mode (default: `false`) |
466+
| `create` | No | Create the database file if it is missing (default: `false`) |
446467

447468
!!! note
448469
SQLite uses Bun's built-in `bun:sqlite` driver. WAL journal mode is enabled automatically for writable databases.
449470

471+
!!! warning "The store must already exist"
472+
As with DuckDB, connecting never creates the database, and a relative `path`
473+
resolves against the directory of the config that declares it — not the current
474+
working directory. See the DuckDB section above for the full rules.
475+
450476
## SQL Server
451477

452478
```json

‎docs/docs/drivers.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,11 @@ MongoDB supports server versions 3.6 through 8.0. Queries use MQL (MongoDB Query
191191
|--------|--------------|
192192
| File | `path: "./my-database.sqlite"` |
193193

194+
For both file-backed drivers, the database must already exist — connecting never
195+
creates it. Pass `create: true` to create it deliberately. A relative `path`
196+
resolves against the directory of the config that declares it, not the current
197+
working directory. See [Warehouses](configure/warehouses.md#duckdb) for the full rules.
198+
194199
## SSH Tunneling
195200

196201
Connect through a bastion host by adding SSH config to any connection:

‎packages/drivers/src/duckdb.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
* DuckDB driver using the `duckdb` package.
33
*/
44

5+
import { assertStoreExists, requireStorePath } from "./file-store"
56
import type { ConnectionConfig, Connector, ConnectorResult, ExecuteOptions, SchemaColumn } from "./types"
67
import { loadOptionalDriver } from "./resolve"
78

@@ -55,7 +56,9 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
5556
duckdb = await loadOptionalDriver("duckdb", "duckdb")
5657
duckdb = duckdb.default || duckdb
5758

58-
const dbPath = (config.path as string) ?? ":memory:"
59+
// altimate_change start — a missing path must fail loudly, not become :memory:
60+
const dbPath = requireStorePath(config, "DuckDB")
61+
// altimate_change end
5962
// altimate_change start — configurable open budget
6063
const { ms: openTimeoutMs, source: openTimeoutSource } = resolveOpenTimeoutMs(config)
6164
// altimate_change end
@@ -118,6 +121,9 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
118121

119122
return {
120123
async connect() {
124+
// altimate_change start — never conjure an empty store on open
125+
assertStoreExists(config, dbPath, "DuckDB")
126+
// altimate_change end
121127
// altimate_change start — retry with read-only on lock errors
122128
const tryConnect = (accessMode?: string): Promise<any> =>
123129
new Promise<any>((resolve, reject) => {

‎packages/drivers/src/file-store.ts‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
/**
2+
* Shared guards for file-backed drivers (DuckDB, SQLite).
3+
*
4+
* Both engines create an empty database when asked to open a file that does
5+
* not exist. For a warehouse connection that is never what the caller wants:
6+
* a mistyped or mis-resolved path then yields a working connector over an
7+
* empty database, so every query succeeds and returns nothing. An agent handed
8+
* that result reports "no tables" instead of an error.
9+
*
10+
* Opening a store is therefore read-or-fail by default. Creation is opt-in via
11+
* `create: true`, which the tools that deliberately materialize a local store
12+
* (local test scratch databases, schema sync targets) pass explicitly.
13+
*/
14+
15+
import * as fs from "fs"
16+
import type { ConnectionConfig } from "./types"
17+
18+
/**
19+
* Whether `dbPath` names a file on the local filesystem, and so can be
20+
* existence-checked before the driver opens it.
21+
*
22+
* Only the exact string `:memory:` is an in-memory database. Both engines
23+
* treat `:memory:named` — and any other colon-prefixed name — as an ordinary
24+
* (if oddly named) file: DuckDB really does write a file called
25+
* `:memory:named`, so those must stay inside the guard. An empty path is
26+
* DuckDB's in-memory database and SQLite's anonymous temporary one; neither
27+
* touches disk.
28+
*
29+
* A scheme-qualified target is not a local file: MotherDuck (`md:`), object
30+
* storage (`s3://`), DuckLake, and any other scheme a DuckDB extension
31+
* provides. Those are left to the driver, which reports an unknown scheme as a
32+
* missing-extension error rather than silently creating anything. The pattern
33+
* requires two or more characters before the colon so a Windows drive letter
34+
* (`C:\data\wh.duckdb`) stays a path.
35+
*/
36+
export function isLocalFilePath(dbPath: string): boolean {
37+
if (dbPath === "" || dbPath === ":memory:") return false
38+
if (/^[a-zA-Z][a-zA-Z0-9+.-]+:/.test(dbPath)) return false
39+
return true
40+
}
41+
42+
/**
43+
* The store path a file-backed connection names, or a loud failure.
44+
*
45+
* Both drivers used to read `(config.path as string) ?? ":memory:"`. That turns
46+
* ANY failure to carry a path — a config the registry never loaded, a field
47+
* under the wrong name, a lookup that fell through — into a successful
48+
* connection over an empty in-memory database. Every query then returns no rows
49+
* and no error, which reads as a healthy warehouse that happens to be empty.
50+
*
51+
* It is a worse failure than creating a store on disk: a stray file can at
52+
* least be found afterwards, whereas an in-memory database leaves nothing
53+
* behind to explain the empty answer. `:memory:` remains available, but only
54+
* when a caller asks for it by name.
55+
*/
56+
export function requireStorePath(config: ConnectionConfig, engine: string): string {
57+
const value = config.path
58+
if (typeof value === "string" && value !== "") return value
59+
throw new Error(
60+
`${engine} connection is missing its "path". A file-backed warehouse must name its database explicitly — ` +
61+
`falling back to an in-memory database would answer every query with no rows and no error, ` +
62+
`which is indistinguishable from a healthy but empty warehouse. ` +
63+
`Set "path" to the database file, or to ":memory:" if a throwaway empty database is genuinely what you want.`,
64+
)
65+
}
66+
67+
/** Whether the caller explicitly opted in to creating the store. */
68+
export function allowsCreate(config: ConnectionConfig): boolean {
69+
return config.create === true
70+
}
71+
72+
/**
73+
* Throw unless the store is safe to open: it already exists, the caller opted
74+
* in to creating it, or the path is not a local file at all.
75+
*
76+
* @param engine Human-readable engine name used in the error message.
77+
* @param allowCreate Whether this open will actually create the store. Defaults
78+
* to the config's `create` flag; a driver passes it explicitly when its own
79+
* options can veto creation — SQLite never creates a read-only connection.
80+
*/
81+
export function assertStoreExists(
82+
config: ConnectionConfig,
83+
dbPath: string,
84+
engine: string,
85+
allowCreate: boolean = allowsCreate(config),
86+
): void {
87+
if (allowCreate) return
88+
if (!isLocalFilePath(dbPath)) return
89+
if (fs.existsSync(dbPath)) return
90+
throw new Error(
91+
`${engine} database file not found: "${dbPath}". ` +
92+
`Opening a warehouse connection never creates the database — an empty store would answer every query with no rows. ` +
93+
`Check the "path" in your connection config (relative paths resolve against the config file's directory, not the current directory), ` +
94+
`or pass "create": true if this store is meant to be created.`,
95+
)
96+
}

‎packages/drivers/src/index.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ export type { Connector, ConnectorResult, SchemaColumn, ConnectionConfig } from
44
// Re-export config normalization
55
export { normalizeConfig, sanitizeConnectionString } from "./normalize"
66

7+
// Re-export file-backed store guards
8+
export { allowsCreate, assertStoreExists, isLocalFilePath, requireStorePath } from "./file-store"
9+
710
// Re-export driver connect functions
811
export { connect as connectPostgres } from "./postgres"
912
export { connect as connectSnowflake } from "./snowflake"

‎packages/drivers/src/sqlite.ts‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,32 @@
44
*/
55

66
import { Database } from "bun:sqlite"
7+
import { allowsCreate, assertStoreExists, requireStorePath } from "./file-store"
78
import type { ConnectionConfig, Connector, ConnectorResult, ExecuteOptions, SchemaColumn } from "./types"
89

910
export async function connect(config: ConnectionConfig): Promise<Connector> {
10-
const dbPath = (config.path as string) ?? ":memory:"
11+
// altimate_change start — a missing path must fail loudly, not become :memory:
12+
const dbPath = requireStorePath(config, "SQLite")
13+
// altimate_change end
1114
let db: Database | null = null
1215

1316
return {
1417
async connect() {
1518
const isReadonly = config.readonly === true
19+
// altimate_change start — never conjure an empty store on open.
20+
// A read-only connection never creates, so `create: true` cannot excuse a
21+
// missing file there; the guard is told the effective decision.
22+
const willCreate = !isReadonly && allowsCreate(config)
23+
assertStoreExists(config, dbPath, "SQLite", willCreate)
1624
db = new Database(dbPath, {
1725
readonly: isReadonly,
18-
create: !isReadonly,
26+
// `create` alone is no longer implied by "not readonly", so the
27+
// read-write flag has to be explicit — bun:sqlite rejects an options
28+
// object that sets no open flag at all.
29+
readwrite: !isReadonly,
30+
create: willCreate,
1931
})
32+
// altimate_change end
2033
if (!isReadonly) {
2134
db.exec("PRAGMA journal_mode = WAL")
2235
}

‎packages/drivers/test/driver-security.test.ts‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,7 @@ describe("DuckDB driver", () => {
178178
}))
179179

180180
const { connect } = await import("../src/duckdb")
181-
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb" })
181+
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb", create: true })
182182
await connector.connect()
183183
expect(connectAttempts).toBe(2) // First failed, second succeeded in READ_ONLY
184184
// The retry must specifically request READ_ONLY — two attempts alone don't
@@ -219,7 +219,11 @@ describe("DuckDB driver", () => {
219219
}))
220220

221221
const { connect } = await import("../src/duckdb")
222-
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb" })
222+
// altimate_change start — bypass the store-existence guard for this mock:
223+
// it never touches the real filesystem, and the test targets lock-error
224+
// classification, not the existence check.
225+
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb", create: true })
226+
// altimate_change end
223227

224228
try {
225229
await connector.connect()
@@ -267,7 +271,10 @@ describe("DuckDB driver", () => {
267271
}))
268272

269273
const { connect } = await import("../src/duckdb")
270-
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb" })
274+
// altimate_change start — bypass the store-existence guard for this mock;
275+
// see the identical note above.
276+
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb", create: true })
277+
// altimate_change end
271278
await connector.connect()
272279

273280
expect(attempts).toBe(2)
@@ -306,7 +313,11 @@ describe("DuckDB driver", () => {
306313
}))
307314

308315
const { connect } = await import("../src/duckdb")
309-
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb", readonly: true })
316+
// altimate_change start — bypass the store-existence guard for this mock;
317+
// see the identical note above. `create` and `readonly` are independent
318+
// flags: `create` only controls the existence check, not access mode.
319+
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb", readonly: true, create: true })
320+
// altimate_change end
310321

311322
try {
312323
await connector.connect()
@@ -405,7 +416,7 @@ describe("DuckDB driver", () => {
405416
}))
406417

407418
const { connect } = await import("../src/duckdb")
408-
const connector = await connect({ type: "duckdb", path: "/tmp/sync-lock.duckdb" })
419+
const connector = await connect({ type: "duckdb", path: "/tmp/sync-lock.duckdb", create: true })
409420
await connector.connect()
410421
expect(connectAttempts).toBe(2)
411422
expect(await connector.execute("SELECT 1")).toMatchObject({ columns: ["ok"], rows: [[1]], row_count: 1 })
@@ -430,7 +441,7 @@ describe("DuckDB driver", () => {
430441
}))
431442

432443
const { connect } = await import("../src/duckdb")
433-
const connector = await connect({ type: "duckdb", path: "/tmp/corrupt.duckdb" })
444+
const connector = await connect({ type: "duckdb", path: "/tmp/corrupt.duckdb", create: true })
434445
await expect(connector.connect()).rejects.toThrow("catalog is corrupt")
435446
})
436447
})

0 commit comments

Comments
 (0)