Skip to content

Commit 3ed4561

Browse files
anandgupta42claude
andauthored
feat: richer column value profiles in schema_inspect (ranked values, uniqueness, value shapes, MongoDB) (#1542)
* feat: richer column value profiles in schema_inspect (ranked values, uniqueness, value shapes, MongoDB paths) - SQL profile (sqlite, duckdb, postgres) stays one query per table but now returns frequency-ranked values with counts; high-cardinality text shows the 3 most frequent values instead of arbitrary ones - mark columns whose distinct count equals their non-null count as `unique in sample` - report the top value shapes of informative text columns (`H01M4/9066` -> `A9+A9/9+`), skipping free text and plain single-shape words - show one ~160 character example for long text columns - MongoDB: the driver reports nested dotted paths (depth <= 3, `field[]` for arrays of objects) with a presence share, exposes `sampleDocuments`, and `schema_inspect` builds the same value profile from the sampled documents Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: [review] MongoDB value profile renders BSON types, dotted keys and MongoDB notation - render UUID, ObjectId, Decimal128, Long, Timestamp, regex, Code and MinKey/MaxKey as readable text; values with no text form report "values not profiled (type)" instead of "all NULL" - carry key paths as segments so keys containing "." are read correctly - name nested fields in MongoDB dot notation (`items.sku`) and mark array membership in a "Nested fields" listing - revert `describeTable` to its top-level behaviour; nested fields are derived in `schema_inspect` from the single document sample used for the value profile - bound the sample with `maxTimeMS` and the tool call's abort signal - add real-BSON unit tests, a describeTable regression test, and a real-server e2e case Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: [review] handle late sampleDocuments rejection and keep array-element marking - attach a handler to the in-flight read so a rejection after abort is not unhandled, and pass the signal to find - accumulate inArray across every occurrence of a nested path so mixed-shape fields are never unique - add tests for both Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: [review] count and rank profile values on the full text, clip only for output SQL groups and ranks the full value and applies SUBSTR in the outer projection; the MongoDB profile keys counts by the full rendered text. Long values sharing a 200 character prefix stay distinct and unique. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 268cbd3 commit 3ed4561

8 files changed

Lines changed: 828 additions & 50 deletions

File tree

‎packages/drivers/src/mongodb.ts‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -389,6 +389,46 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
389389
return JSON.stringify(val)
390390
}
391391

392+
/**
393+
* Raw documents for value profiling. `maxTimeMS` is the smaller of the caller's cap and the
394+
* connection's query timeout; `signal` abandons the read and closes the cursor.
395+
*/
396+
async function sampleDocuments(
397+
schema: string,
398+
table: string,
399+
limit = 100,
400+
options: { signal?: AbortSignal; maxTimeMS?: number } = {},
401+
): Promise<Record<string, unknown>[]> {
402+
options.signal?.throwIfAborted()
403+
const configured = resolveQueryTimeout(config).ms
404+
const maxTimeMS =
405+
options.maxTimeMS === undefined ? configured : configured === undefined ? options.maxTimeMS : Math.min(configured, options.maxTimeMS)
406+
const { signal } = options
407+
// The driver (6.x) cancels the operation on `signal`; the race below also covers fakes and older drivers.
408+
const cursor = client
409+
.db(schema)
410+
.collection(table)
411+
.find({}, { ...withMaxTime(maxTimeMS), ...(signal ? { signal } : {}) })
412+
.limit(limit)
413+
if (!signal) return cursor.toArray()
414+
let abort: (() => void) | undefined
415+
const aborted = new Promise<never>((_, reject) => {
416+
abort = () => {
417+
void Promise.resolve(cursor.close?.()).catch(() => {})
418+
reject(signal.reason)
419+
}
420+
signal.addEventListener("abort", abort, { once: true })
421+
})
422+
// A read that rejects after the abort won the race must not surface as an unhandled rejection.
423+
const read: Promise<Record<string, unknown>[]> = cursor.toArray()
424+
read.catch(() => {})
425+
try {
426+
return await Promise.race([read, aborted])
427+
} finally {
428+
signal.removeEventListener("abort", abort!)
429+
}
430+
}
431+
392432
const connector: Connector & {
393433
// altimate_change start
394434
runCommand(
@@ -910,6 +950,8 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
910950
.sort((a: { name: string }, b: { name: string }) => a.name.localeCompare(b.name))
911951
},
912952

953+
sampleDocuments,
954+
913955
async describeTable(schema: string, table: string): Promise<SchemaColumn[]> {
914956
const db = client.db(schema)
915957
const coll = db.collection(table)

‎packages/drivers/src/types.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,13 @@ export interface Connector {
135135
listSchemas(): Promise<string[]>
136136
listTables(schema: string): Promise<Array<{ name: string; type: string }>>
137137
describeTable(schema: string, table: string): Promise<SchemaColumn[]>
138+
/** Document stores only: raw documents, bounded by `maxTimeMS` and cancellable through `signal`. */
139+
sampleDocuments?(
140+
schema: string,
141+
table: string,
142+
limit?: number,
143+
options?: { signal?: AbortSignal; maxTimeMS?: number },
144+
): Promise<Record<string, unknown>[]>
138145
introspectSchema?(signal?: AbortSignal): Promise<SchemaMetadata>
139146
close(): Promise<void>
140147
}

‎packages/drivers/test/mongodb-unit.test.ts‎

Lines changed: 57 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,10 @@ const { Binary } = createRequire(import.meta.resolve("mongodb"))("bson")
1010

1111
async function fakeMongo(
1212
docs?: Record<string, unknown>[],
13-
metadata: { collections?: Record<string, string[]>; error?: Error } = {},
13+
metadata: { collections?: Record<string, string[]>; error?: Error; read?: () => Promise<Record<string, unknown>[]> } = {},
1414
) {
1515
const finds: unknown[] = []
16+
const findOptions: unknown[] = []
1617
const pipelines: Record<string, unknown>[][] = []
1718
const aggregateOptions: unknown[] = []
1819
const counts: { filter: unknown; options: unknown }[] = []
@@ -50,9 +51,10 @@ async function fakeMongo(
5051
collections[database].push(name)
5152
},
5253
collection: () => ({
53-
find: (filter: unknown) => {
54+
find: (filter: unknown, options: unknown) => {
5455
finds.push(filter)
55-
return { limit: () => ({ toArray: async () => docs ?? [], async *[Symbol.asyncIterator]() { yield* docs ?? [] } }) }
56+
findOptions.push(options)
57+
return { limit: () => ({ toArray: () => (metadata.read ? metadata.read() : Promise.resolve(docs ?? [])), async *[Symbol.asyncIterator]() { yield* docs ?? [] } }) }
5658
},
5759
aggregate: (pipeline: Record<string, unknown>[], options: unknown) => {
5860
pipelines.push(pipeline)
@@ -81,7 +83,7 @@ async function fakeMongo(
8183
const connector = await connect({ type: "mongodb", database: "test" })
8284
await connector.connect()
8385
const query = (command: string, extra: object = {}) => JSON.stringify({ collection: "articles", command, ...extra })
84-
return { connector, query, collections, finds, pipelines, aggregateOptions, counts, distincts, collectionLists, databaseLists, estimates }
86+
return { connector, query, collections, finds, findOptions, pipelines, aggregateOptions, counts, distincts, collectionLists, databaseLists, estimates }
8587
}
8688

8789
describe("MongoDB read command shapes", () => {
@@ -473,6 +475,57 @@ describe("MongoDB native command filters", () => {
473475
})
474476
})
475477

478+
describe("MongoDB nested documents", () => {
479+
const docs = [
480+
{ _id: 1, user: { name: "a", geo: { lat: 1.5 } }, items: [{ sku: "x" }], "a.b": 1 },
481+
{ _id: 2, user: null, items: [] },
482+
]
483+
484+
test("describeTable stays top-level: nested objects are 'object', arrays 'array', no nested paths", async () => {
485+
const { connector } = await fakeMongo(docs)
486+
expect(await connector.describeTable("test", "articles")).toEqual([
487+
{ name: "_id", data_type: "int32", nullable: false },
488+
{ name: "a.b", data_type: "int32", nullable: true },
489+
{ name: "items", data_type: "array", nullable: false },
490+
{ name: "user", data_type: "object", nullable: true },
491+
])
492+
})
493+
494+
test("sampleDocuments returns raw documents under the smaller of the caller and connection deadlines", async () => {
495+
const { connector, findOptions } = await fakeMongo(docs)
496+
expect(await connector.sampleDocuments!("test", "articles", 50, { maxTimeMS: 15_000 })).toEqual(docs)
497+
expect(findOptions).toEqual([{ maxTimeMS: 15_000 }])
498+
})
499+
500+
test("sampleDocuments honours an aborted signal", async () => {
501+
const { connector } = await fakeMongo(docs)
502+
await expect(connector.sampleDocuments!("test", "articles", 50, { signal: AbortSignal.abort(new Error("stop")) })).rejects.toThrow("stop")
503+
})
504+
505+
test("a read rejecting after the abort is handled and the abort reason wins", async () => {
506+
const unhandled: unknown[] = []
507+
const onUnhandled = (reason: unknown) => unhandled.push(reason)
508+
process.on("unhandledRejection", onUnhandled)
509+
try {
510+
let rejectRead!: (e: Error) => void
511+
const { connector, findOptions } = await fakeMongo(docs, {
512+
read: () => new Promise((_, reject) => { rejectRead = reject }),
513+
})
514+
const controller = new AbortController()
515+
const pending = connector.sampleDocuments!("test", "articles", 50, { signal: controller.signal })
516+
const abortReason = new Error("cancelled")
517+
controller.abort(abortReason)
518+
await expect(pending).rejects.toBe(abortReason)
519+
rejectRead(new Error("cursor closed"))
520+
await new Promise((resolve) => setTimeout(resolve, 20))
521+
expect(unhandled).toEqual([])
522+
expect(findOptions[0]).toMatchObject({ signal: controller.signal })
523+
} finally {
524+
process.off("unhandledRejection", onUnhandled)
525+
}
526+
})
527+
})
528+
476529
describe("MongoDB collection command shorthand", () => {
477530
test.each([
478531
{ name: "key", aliases: { key: "status" } },

‎packages/opencode/src/altimate/tools/schema-inspect.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,9 @@ export const SchemaInspectTool = Tool.define("schema_inspect", {
128128
table: ref,
129129
columns: schemaResult.columns!,
130130
warehouse: schemaResult.warehouse ?? target.warehouse,
131+
...(schemaResult.schema_name
132+
? { document: { database: schemaResult.schema_name, collection: schemaResult.table ?? args.table } }
133+
: {}),
131134
},
132135
ctx,
133136
)

0 commit comments

Comments
 (0)