Skip to content

Commit e0d5dec

Browse files
anandgupta42claude
andcommitted
fix(drivers): classify a locked DuckDB store, and stop treating a lock as a broken install
Review round on #1198. Fifteen threads, four of them the same defect. **A lock on an explicitly read-only open was reported as an unclassified failure.** `connect()` only wrapped a lock error on the retry path, and an explicit `config.readonly` open never reaches that path — correctly, since DuckDB's file lock is exclusive against read-only opens too, so retrying `READ_ONLY` when we already asked for `READ_ONLY` would just repeat the same failure. But the `else` rethrew DuckDB's raw text, and `categorizeConnectionError` matches only the wrapper's `locked by another process` wording, so a plain lock collision came out as `other` — the same shape as a wrong password. Now wrapped (still not retried). `:memory:` is excluded for the same reason the retry excludes it: no other process can hold it, so the wrapper's text would be false. **`categorizeConnectionError` now also matches DuckDB's raw text** — `conflicting lock` and `could not set lock` — so a lock that reaches it unwrapped from any other path is still classified. **A locked store is no longer rendered as a broken client.** It stays `infrastructure` (nothing about the connection's config is wrong, and a harness must not score it as a task failure) but is now also `recoverable`, and `warehouse_test` renders it as `STORE LOCKED` with the remedy the driver's own message already gives — close the conflicting connection and retry — instead of the stop-and-report copy meant for an install that will never work. **Also fixed** - `setTimeout` clamps a delay past 2^31-1 to about 1ms, so a deliberately huge `open_timeout_ms` became an immediate deadline — the exact inversion this change exists to prevent. Clamped before it reaches the timer. - The deadline message named only `ALTIMATE_DUCKDB_OPEN_TIMEOUT_MS`, which `config.open_timeout_ms` overrides. It now names both, and which one wins. - A deadline that fires because the open callback never arrives left the native handle open. When the callback does arrive late, `onOpen` closes it; when it never arrives that branch never runs, so the timeout closes it too. Both paths are idempotent. - Dropped a dead `msg.includes("duckdb_locked")` clause: `msg` is lowercased, so `locked` already matches it. - The `drivers` CI path filter did not list the two DuckDB E2E files its own step runs, so a PR touching only those files got no execution anywhere — the driver job skipped, and the main job runs them with the gate unset. - `Registry.reset()` is synchronous and `close()` is not, so resetting leaked a native handle per connector. Added `Registry.closeAll()`, used by `reload()` and by the E2E suite. - The E2E suite deleted `ALTIMATE_TELEMETRY_DISABLED` on teardown instead of restoring what it found. - `bun.lock` did not record the `trustedDependencies` addition. **Tests** - `driver-security.test.ts`: an explicitly read-only open that hits a lock is wrapped, and attempted exactly once with `READ_ONLY`. Fails without the fix. - `connections.test.ts`: eight cases pinning the classification — DuckDB's verbatim lock text (which contains `lock` but never `locked`), the wrapper's text, the open deadline, and which categories are infrastructure vs recoverable. These run in the normal suite, ungated. - `duckdb-lock-helper.ts`: holds a write lock from a separate OS process. In-process is not a test — DuckDB's lock is per-process. - E2E: read-only and read-write opens against a foreign lock both report a lock and keep DuckDB's PID detail; the store is usable once the lock is released (which guards the helper itself); an over-large budget does not become 1ms; `warehouse_test` reports `store_locked` + recoverable and renders `STORE LOCKED` rather than stop-and-report. Gates: typecheck 13/13 (it caught a real error in the new helper, so it does cover `packages/opencode/test`); markers ok against `origin/main`; drivers 141 pass 0 fail; `test/altimate` 4222 pass 0 fail; gated DuckDB E2E 19 pass 0 fail. Lint is 5863 warnings / 1 error: the error is the pre-existing `consistent-return` in `packages/http-recorder/test/record-replay.test.ts`, and the warning count is +1 from an un-awaited `mock.module` in the new test, which is how every other `mock.module` in that file is written. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ
1 parent e9466da commit e0d5dec

11 files changed

Lines changed: 440 additions & 15 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,14 @@ jobs:
6060
- 'packages/opencode/test/altimate/drivers-mongodb-e2e.test.ts'
6161
- 'packages/opencode/test/altimate/drivers-clickhouse-e2e.test.ts'
6262
- 'packages/opencode/test/altimate/connections.test.ts'
63+
# Run by the "DuckDB store-open E2E" step in the driver-e2e job.
64+
# Without these, a PR touching only these files skips that job,
65+
# and the main TypeScript job runs them with ALTIMATE_DUCKDB_E2E
66+
# unset, which skips every test in them — no execution anywhere.
67+
- 'packages/opencode/test/altimate/duckdb-open-e2e.test.ts'
68+
- 'packages/opencode/test/altimate/warehouse-test-duckdb-e2e.test.ts'
69+
- 'packages/opencode/test/altimate/duckdb-lock-helper.ts'
70+
- 'packages/opencode/src/altimate/tools/warehouse-test.ts'
6371
dbt-tools:
6472
- 'packages/dbt-tools/**'
6573
installer:

‎bun.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/drivers/src/duckdb.ts‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,19 @@ import type { ConnectionConfig, Connector, ConnectorResult, ExecuteOptions, Sche
1717
*/
1818
const DEFAULT_OPEN_TIMEOUT_MS = 30_000
1919

20+
/**
21+
* Largest delay `setTimeout` represents. A larger value overflows the timer's
22+
* 32-bit signed delay and is clamped to 1ms, which would turn a deliberately
23+
* huge budget into an immediate deadline — the exact failure this file exists
24+
* to remove. Clamp instead, so an over-large budget still behaves like a very
25+
* long one.
26+
*/
27+
const MAX_TIMER_MS = 2_147_483_647
28+
2029
/** Read a positive, finite millisecond budget, or `undefined` if unusable. */
2130
function positiveMs(value: unknown): number | undefined {
2231
const n = typeof value === "number" ? value : Number(value)
23-
return Number.isFinite(n) && n > 0 ? n : undefined
32+
return Number.isFinite(n) && n > 0 ? Math.min(n, MAX_TIMER_MS) : undefined
2433
}
2534

2635
function resolveOpenTimeoutMs(config: ConnectionConfig): number {
@@ -59,8 +68,7 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
5968
msg.includes("locked") ||
6069
msg.includes("conflicting lock") ||
6170
msg.includes("could not set lock") ||
62-
msg.includes("sqlite_busy") ||
63-
msg.includes("duckdb_locked")
71+
msg.includes("sqlite_busy")
6472
)
6573
}
6674

@@ -157,12 +165,19 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
157165
timeout = setTimeout(() => {
158166
if (!resolved) {
159167
resolved = true
168+
// Nothing can reach this handle once the promise rejects. If the
169+
// callback arrives later it closes the handle itself (see the
170+
// `resolved` branch in onOpen); if it never arrives, that branch
171+
// never runs, so close here too. Both paths are idempotent
172+
// because closeQuietly swallows a double close.
173+
closeQuietly()
160174
reject(
161175
new Error(
162176
`DuckDB store "${dbPath}" did not finish opening within ${openTimeoutMs}ms. ` +
163177
`This is a client-side deadline in the DuckDB driver, not a fault in the store ` +
164178
`— the open may simply be queued behind other work in this process. ` +
165-
`Raise it with ALTIMATE_DUCKDB_OPEN_TIMEOUT_MS if the machine is loaded.`,
179+
`Raise the budget with this connection's open_timeout_ms (which takes ` +
180+
`priority) or ALTIMATE_DUCKDB_OPEN_TIMEOUT_MS if the machine is loaded.`,
166181
),
167182
)
168183
}
@@ -192,6 +207,17 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
192207
retryErr instanceof Error ? retryErr : new Error(String(retryErr)),
193208
)
194209
}
210+
} else if (isLockError(err) && dbPath !== ":memory:") {
211+
// An explicit read-only open is NOT rescued by the retry above — and
212+
// must not be: DuckDB's file lock is exclusive against read-only
213+
// opens too, so re-opening READ_ONLY when we already asked for
214+
// READ_ONLY would only repeat the same failure. It still has to be
215+
// wrapped, because categorizeConnectionError matches the wrapper's
216+
// "locked by another process" wording; the raw DuckDB text would be
217+
// reported as an unclassified failure. An in-memory store is excluded
218+
// for the same reason the retry excludes it: no other process can
219+
// hold it, so the wrapper's text would be false.
220+
throw wrapDuckDBError(err instanceof Error ? err : new Error(String(err)))
195221
} else {
196222
throw err
197223
}

‎packages/drivers/test/driver-security.test.ts‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,13 @@ describe("DuckDB driver", () => {
136136
})
137137
})
138138

139+
// Verbatim DuckDB output for a cross-process lock collision. It contains
140+
// "lock" but never "locked", which is why the driver's original
141+
// `.includes("locked")` check never matched a real collision.
142+
const REAL_DUCKDB_LOCK_ERROR =
143+
'IO Error: Could not set lock on file "/tmp/test.duckdb": Conflicting lock is held in ' +
144+
"/usr/bin/node (PID 65001) by user someone."
145+
139146
describe("connect retry with READ_ONLY", () => {
140147
test("retries with READ_ONLY when file DB is locked on initial connect", async () => {
141148
let connectAttempts = 0
@@ -182,6 +189,51 @@ describe("DuckDB driver", () => {
182189
await connector.close()
183190
})
184191

192+
test("wraps a lock error on an explicitly read-only open, and does not retry it", async () => {
193+
// A caller that set `readonly` already gets READ_ONLY on the first open,
194+
// so there is nothing for the retry to try differently — DuckDB's file
195+
// lock is exclusive against read-only opens too. The failure must still
196+
// be *labelled* a lock, though: consumers classify it by the wrapper's
197+
// "locked by another process" wording, and this branch used to rethrow
198+
// DuckDB's raw text, which contains "lock" but never "locked".
199+
let attempts = 0
200+
const accessModes: Array<string | undefined> = []
201+
mock.module("duckdb", () => ({
202+
default: {
203+
Database: class {
204+
constructor(_path: string, optsOrCb: any, cb?: (err: Error | null) => void) {
205+
const opts = typeof optsOrCb === "function" ? undefined : optsOrCb
206+
accessModes.push(opts?.access_mode)
207+
attempts++
208+
setTimeout(() => openCallback(optsOrCb, cb)(new Error(REAL_DUCKDB_LOCK_ERROR)), 0)
209+
}
210+
connect() {
211+
return {}
212+
}
213+
close(cb: any) {
214+
if (cb) cb(null)
215+
}
216+
},
217+
},
218+
}))
219+
220+
const { connect } = await import("../src/duckdb")
221+
const connector = await connect({ type: "duckdb", path: "/tmp/test.duckdb", readonly: true })
222+
223+
try {
224+
await connector.connect()
225+
expect.unreachable("Should have thrown")
226+
} catch (e: any) {
227+
expect(e.message).toContain("locked by another process")
228+
// DuckDB's own text survives: it names the PID holding the lock.
229+
expect(e.message).toContain("Conflicting lock is held")
230+
}
231+
232+
// Exactly one attempt, and it asked for READ_ONLY up front.
233+
expect(attempts).toBe(1)
234+
expect(accessModes).toEqual(["READ_ONLY"])
235+
})
236+
185237
test("does not retry in-memory DB on lock error", async () => {
186238
mock.module("duckdb", () => ({
187239
default: {

‎packages/opencode/src/altimate/native/connections/registry.ts‎

Lines changed: 48 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -286,7 +286,16 @@ export function categorizeConnectionError(e: unknown): string {
286286
// Checked before the generic "timeout"/"not found" rules below, which are
287287
// about the remote warehouse and would otherwise swallow these.
288288
if (msg.includes("did not finish opening")) return "driver_open_timeout"
289-
if (msg.includes("locked by another process")) return "store_locked"
289+
// "locked by another process" is the DuckDB driver's own wrapper. The other
290+
// two are DuckDB's raw text ("Could not set lock on file …: Conflicting lock
291+
// is held"), matched here as well so a lock that reaches this function
292+
// unwrapped is still classified rather than falling through to "other".
293+
if (
294+
msg.includes("locked by another process") ||
295+
msg.includes("conflicting lock") ||
296+
msg.includes("could not set lock")
297+
)
298+
return "store_locked"
290299
// altimate_change end
291300
if (msg.includes("password") || msg.includes("authentication") || msg.includes("unauthorized") || msg.includes("jwt"))
292301
return "auth_failed"
@@ -298,15 +307,27 @@ export function categorizeConnectionError(e: unknown): string {
298307

299308
// altimate_change start — distinguish infrastructure faults from config faults
300309
/**
301-
* Categories where the local client is broken, not the connection's config and
302-
* not the remote warehouse. A caller cannot fix these by correcting
310+
* Categories where the fault is in this machine, not the connection's config
311+
* and not the remote warehouse. A caller cannot fix these by correcting
303312
* credentials, and a harness must not score them as a task failure.
304313
*/
305314
const INFRASTRUCTURE_CATEGORIES = new Set(["driver_missing", "driver_open_timeout", "store_locked"])
306315

316+
/**
317+
* The subset of the above that clears on its own once another process lets go.
318+
* Still infrastructure — nothing about the connection's config is wrong — but
319+
* the right response is to close the conflicting connection and retry, not to
320+
* stop and report a broken install.
321+
*/
322+
const RECOVERABLE_CATEGORIES = new Set(["store_locked"])
323+
307324
export function isInfrastructureFailure(category: string): boolean {
308325
return INFRASTRUCTURE_CATEGORIES.has(category)
309326
}
327+
328+
export function isRecoverableFailure(category: string): boolean {
329+
return RECOVERABLE_CATEGORIES.has(category)
330+
}
310331
// altimate_change end
311332

312333
// ---------------------------------------------------------------------------
@@ -426,7 +447,13 @@ export function list(): { warehouses: WarehouseInfo[] } {
426447
/** Test a connection by running a simple query. */
427448
export async function test(
428449
name: string,
429-
): Promise<{ connected: boolean; error?: string; error_category?: string; infrastructure?: boolean }> {
450+
): Promise<{
451+
connected: boolean
452+
error?: string
453+
error_category?: string
454+
infrastructure?: boolean
455+
recoverable?: boolean
456+
}> {
430457
try {
431458
const connector = await get(name)
432459
const config = configs.get(name)
@@ -450,6 +477,7 @@ export async function test(
450477
error: String(e),
451478
error_category: category,
452479
infrastructure: isInfrastructureFailure(category),
480+
recoverable: isRecoverableFailure(category),
453481
}
454482
// altimate_change end
455483
}
@@ -545,9 +573,16 @@ export async function remove(name: string): Promise<{ success: boolean; error?:
545573
}
546574
}
547575

548-
/** Reload all configs and clear cached connectors. */
549-
export async function reload(): Promise<void> {
550-
// Close all cached connectors
576+
// altimate_change start — a way to release native handles without reloading
577+
/**
578+
* Close every cached connector and drop it from the cache.
579+
*
580+
* `reset()` cannot do this: it is synchronous, and `close()` is not. Callers
581+
* that create real connectors (tests, and `reload()` below) must await this,
582+
* otherwise the native handle behind each connector stays open — on Windows
583+
* that also blocks deleting the file underneath it.
584+
*/
585+
export async function closeAll(): Promise<void> {
551586
for (const [, connector] of connectors) {
552587
try {
553588
await connector.close()
@@ -556,6 +591,12 @@ export async function reload(): Promise<void> {
556591
}
557592
}
558593
connectors.clear()
594+
}
595+
// altimate_change end
596+
597+
/** Reload all configs and clear cached connectors. */
598+
export async function reload(): Promise<void> {
599+
await closeAll()
559600
loaded = false
560601
load()
561602
}

‎packages/opencode/src/altimate/native/types.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -339,6 +339,12 @@ export interface WarehouseTestResult {
339339
* are not fixable by editing the connection.
340340
*/
341341
infrastructure?: boolean
342+
/**
343+
* True when the failure clears on its own once another process lets go — a
344+
* store locked by another writer. Still `infrastructure`, but the response is
345+
* to close the conflicting connection and retry, not to stop and report.
346+
*/
347+
recoverable?: boolean
342348
// altimate_change end
343349
}
344350

‎packages/opencode/src/altimate/tools/warehouse-test.ts‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,33 @@ export const WarehouseTestTool = Tool.define("warehouse_test", {
2525
// fault in this machine's install. Reporting it with the same wording as
2626
// a wrong password invites both the model and anyone reading the
2727
// transcript to treat broken infrastructure as a task or config failure.
28+
// A locked store is infrastructure too — nothing about the connection's
29+
// config is wrong — but it is the one category that clears itself once
30+
// the other process lets go, and the driver's own message names the PID
31+
// holding the lock. Telling the model to stop and report would be wrong
32+
// advice for the one fault it can actually act on.
33+
if (result.recoverable) {
34+
return {
35+
title: `Connection '${args.name}': STORE LOCKED`,
36+
metadata: {
37+
connected: false,
38+
error: result.error,
39+
error_category: result.error_category,
40+
infrastructure: true,
41+
recoverable: true,
42+
},
43+
output:
44+
`STORE LOCKED — another process holds this store open, so it could not be ` +
45+
`opened here. This is NOT a problem with the connection's configuration and ` +
46+
`NOT a credentials failure.\n` +
47+
`Category: ${result.error_category}\n` +
48+
`Error: ${result.error ?? "Unknown error"}\n\n` +
49+
`The error above names the process holding the lock. Close that connection ` +
50+
`and try again. Do not work around this by querying something else — until ` +
51+
`the lock clears, no result here comes from warehouse '${args.name}'.`,
52+
}
53+
}
54+
2855
if (result.infrastructure) {
2956
return {
3057
title: `Connection '${args.name}': INFRASTRUCTURE FAILURE`,

‎packages/opencode/test/altimate/connections.test.ts‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,12 @@ afterAll(() => { delete process.env.ALTIMATE_TELEMETRY_DISABLED })
1010
// ---------------------------------------------------------------------------
1111

1212
import * as Registry from "../../src/altimate/native/connections/registry"
13-
import { detectAuthMethod } from "../../src/altimate/native/connections/registry"
13+
import {
14+
categorizeConnectionError,
15+
detectAuthMethod,
16+
isInfrastructureFailure,
17+
isRecoverableFailure,
18+
} from "../../src/altimate/native/connections/registry"
1419
import * as CredentialStore from "../../src/altimate/native/connections/credential-store"
1520
import { parseDbtProfiles, dbtConnectionsToConfigs } from "../../src/altimate/native/connections/dbt-profiles"
1621
import { discoverContainers, containerToConfig } from "../../src/altimate/native/connections/docker-discovery"
@@ -20,6 +25,57 @@ import { registerAll } from "../../src/altimate/native/connections/register"
2025
// ConnectionRegistry
2126
// ---------------------------------------------------------------------------
2227

28+
// altimate_change start — classification of local-client faults
29+
describe("categorizeConnectionError: local-client faults", () => {
30+
// Verbatim DuckDB output. It contains "lock" but never "locked", which is
31+
// why matching only the driver's own wrapper wording missed it.
32+
const RAW_DUCKDB_LOCK =
33+
'IO Error: Could not set lock on file "/tmp/warehouse.duckdb": Conflicting lock is held in ' +
34+
"/usr/bin/node (PID 65001) by user someone. See also https://duckdb.org/docs/stable/connect/concurrency"
35+
36+
test("the raw DuckDB lock message never contains the substring 'locked'", () => {
37+
expect(RAW_DUCKDB_LOCK.toLowerCase().includes("locked")).toBe(false)
38+
})
39+
40+
test("classifies the raw DuckDB lock message as store_locked", () => {
41+
expect(categorizeConnectionError(new Error(RAW_DUCKDB_LOCK))).toBe("store_locked")
42+
})
43+
44+
test("classifies the driver's wrapped lock message as store_locked", () => {
45+
const wrapped = new Error(`Database "/tmp/w.duckdb" is locked by another process. ${RAW_DUCKDB_LOCK}`)
46+
expect(categorizeConnectionError(wrapped)).toBe("store_locked")
47+
})
48+
49+
test("classifies the open deadline as driver_open_timeout, not a remote timeout", () => {
50+
const err = new Error('DuckDB store "/tmp/w.duckdb" did not finish opening within 30000ms.')
51+
expect(categorizeConnectionError(err)).toBe("driver_open_timeout")
52+
})
53+
54+
test("still classifies a real credentials failure as auth_failed", () => {
55+
expect(categorizeConnectionError(new Error("password authentication failed for user"))).toBe("auth_failed")
56+
})
57+
58+
test("a locked store is infrastructure, and is the one that is recoverable", () => {
59+
expect(isInfrastructureFailure("store_locked")).toBe(true)
60+
expect(isRecoverableFailure("store_locked")).toBe(true)
61+
})
62+
63+
test("a broken install is infrastructure but NOT recoverable", () => {
64+
for (const category of ["driver_missing", "driver_open_timeout"]) {
65+
expect(isInfrastructureFailure(category)).toBe(true)
66+
expect(isRecoverableFailure(category)).toBe(false)
67+
}
68+
})
69+
70+
test("a config or credentials fault is neither infrastructure nor recoverable", () => {
71+
for (const category of ["auth_failed", "config_error", "network_error", "other"]) {
72+
expect(isInfrastructureFailure(category)).toBe(false)
73+
expect(isRecoverableFailure(category)).toBe(false)
74+
}
75+
})
76+
})
77+
// altimate_change end
78+
2379
describe("ConnectionRegistry", () => {
2480
beforeEach(() => {
2581
Registry.reset()

0 commit comments

Comments
 (0)