Skip to content

Commit e0dc3da

Browse files
hkaltaiclaude
andauthored
feat: structured lineage and column reach for dbt_lineage and impact_analysis (#1635)
* feat: structured lineage and column reach for dbt_lineage and impact_analysis Builds on the dbt integration's structured dbt_lineage result. - dbt_lineage takes a project-relative SQL path or a snapshot as well as a model, and with include_graph returns the model's graph from the manifest (two levels up, everything downstream with exposures and tests, capped at 500 nodes). The manifest fallback reports upstream ids, column mappings and edges as data. - impact_analysis with a column traces each downstream model's and snapshot's compiled SQL in dependency order and reports column_reach: certain readers with the columns they carry, possible readers (filters and joins, select * over undocumented columns, shared table names, ephemeral parents), untraced models and the exposures on each. It matches models by unique_id and honours the dialect override. - The MCP server declares an output schema for both tools. Structured content is the tool's own result when it builds one (the dbt integration's model lineage), else its lineage fields, always with the text report and capped at 100 KB, errors included. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: dbt_lineage uses the adapter only for the named manifest's own project Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: an oversized structured result keeps its report within the cap and stays marked truncated Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs: the truncated flag says the report may be shortened too Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat: dbt_lineage include_graph returns the model's whole upstream, capped at 500 nodes and marked when cut Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: include_graph leaves out the tests of upstream models Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: address review comments on lineage structured results Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a column spelling that matches two columns differing only by case leaves their readers uncertain Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: an unsettled column spelling leaves the model's own dashboards uncertain, and project containment compares whole path segments Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: uncertainty for models the engine cannot fully trace, a token-safe structured cap, header-only adapter report, dashboards through metrics Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a model with no column edges is uncertain only when its SQL names the column Co-Authored-By: Claude Opus 5.5 <[email protected]> * test: a model below a partly traced reader is a possible reader Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a structured result passed through for a tool outside the declared set is held to the same cap Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor: the column-name scan runs once per model; test rules out a certain reader for a constant model Co-Authored-By: Claude Opus 5.5 <[email protected]> * style: indent the passthrough condition as Prettier does Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor: drop the structured passthrough no tool reaches; dbt_lineage takes the declared, capped path Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: an uncompiled project's own manifest path keeps the dbt integration's answer, and whole-row reads are possible reads - a manifest_path not written yet is placed by its path (nearest existing folder's links resolved): inside the adapter's project it keeps the adapter's lineage; elsewhere it is still another project's - a parent's row used as a value (to_json(o), cast(o as varchar)) or a NATURAL JOIN makes a model a possible reader, never a confident non-reader - downstream entries carry unique_id; a SQL path matches with ./ or backslashes - count_big(*)/count_if(*) are not a select *; a backtick output alias is not a read - include_graph's description says the graph is left out of MCP structured results Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a whole row is a row source passed whole to a call, so import CTEs stay confident and quoted, CTE and subquery rows are caught - row sources: FROM/JOIN relations and aliases (quoted or not, never a keyword), CTE names and subquery aliases - a whole row read beside the column passes the doubt on to the model's children - a missing manifest with no dbt project above it names no other project; a link to a missing manifest cannot be placed Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: column reach flags a NATURAL JOIN only, and a manifest link is placed where it points A pattern cannot tell a row source from a column (sum(x) as x, extract(year from d)), so whole-row values are left to the lineage engine; a NATURAL JOIN is one keyword with no such ambiguity. A link to a manifest not written yet is followed, and a path through a file is placed instead of throwing. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: every NATURAL join form counts and passes its doubt on, quoted names never do, and a relative link climbs from its real folder Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a NATURAL JOIN inside array brackets or after an escaped quote still counts Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a NATURAL JOIN is found in the raw SQL, so no dialect's quoting can hide one Masking strings and quoted names to avoid a false positive let quoting rules that differ by dialect hide real joins. The words inside a string or comment now count too, which only adds doubt. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: comments between NATURAL and JOIN are read as gaps, with no pass over the whole SQL Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: NATURAL JOIN lookup reads // and CR-ended comments, and stays bounded on any input Each natural word is checked in a 1000-character window, and past 100 of them the SQL counts as having one, which only adds doubt. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: Snowflake's DIRECTED NATURAL JOIN counts too Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a manifest is placed where the system opens it, through Filesystem.realLocation; an empty path is no model's; NATURAL JOIN gaps up to 10,000 characters - links resolve before `..`, so `<project>/link/../x` lands where the link points; the duplicate resolver is gone, and a link loop cannot be placed - a request that is empty once normalized never matches a model without a file path - the window per natural grows to 10,000 characters, still bounded with the 100-word cap - tests for count_big(*)/count_if(*), a backticked output alias, link/.. and the empty path Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a manifest is placed as the manifest reader opens it, and only link hops count toward the loop limit loadRawManifest opens a path through realpathSync, which collapses `..` before following links, so the project check does the same: the adapter's lineage and the graph read beside it always come from one project. Climbing to the nearest existing folder no longer counts toward the 64-link limit. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor: the requested SQL path is normalized once, and the link/.. test proves which manifest the graph reads Co-Authored-By: Claude Opus 5.5 <[email protected]> * style: Prettier on the files this change left unformatted Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a column named through Snowflake IDENTIFIER() is a possible read, and a column reach that stopped early reports lower bounds - IDENTIFIER('col') with a traced column, or a bind variable, makes the model uncertain and passes the doubt on, as a NATURAL JOIN does - the text report says "at least" when column reach stopped early, and the exposures past the stop are unchecked, not non-readers - the model parameter says a snapshot is found by name too Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: comments before or inside IDENTIFIER( are read as gaps, and its lookup is bounded like the NATURAL JOIN one Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: any mention of a traced column inside IDENTIFIER('…') counts, so a quoted name with a dot is never split Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a doubled quote inside an IDENTIFIER() name is read as one quote Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: an IDENTIFIER() string that is not a plain name counts as naming the column, so no escape can hide one Escapes (\x72), backslash-escaped quotes and other characters are not decoded; such a string marks the model uncertain, which only adds doubt. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: only an IDENTIFIER() whose whole argument is one plain string naming another column is cleared Concatenations, bind and session variables, functions and unterminated strings all count as possibly naming the column, so no argument form can produce a confident zero. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: a column reach that stopped early bounds every count, and the other exposures are not all called unchecked - uncertain and untraced counts say "at least" too when column reach stopped early - the other exposures' header says some of them were not checked, after the count - the parents' traced columns are built once Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs: the stopped-early exposures note says some may not have been checked Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent a6412db commit e0dc3da

13 files changed

Lines changed: 2259 additions & 84 deletions

File tree

‎packages/opencode/src/altimate/mcp-server/server.ts‎

Lines changed: 82 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,81 @@ export async function serverInstructions(): Promise<string> {
4343
}
4444
}
4545

46+
// altimate_change start — lineage tools answer with structured content as well as text
47+
// Lineage tools also return their metadata as structured content, so a client that draws lineage
48+
// (the Claude Code plugin's lineage pane) reads edges and downstream lists without parsing text.
49+
// Claude Code shows structured content to the model in place of the text, so `report` carries the
50+
// (already truncated) text (dbt_lineage: its header, its mappings travel as data), only the lineage
51+
// fields are passed, and the whole is capped.
52+
const STRUCTURED_FIELDS: Record<string, readonly string[]> = {
53+
dbt_lineage: ["model_name", "model_unique_id", "confidence", "confidence_factors", "upstream", "columns", "edges", "error"],
54+
impact_analysis: [
55+
"model",
56+
"column",
57+
"change_type",
58+
"severity",
59+
"direct_count",
60+
"transitive_count",
61+
"test_count",
62+
"downstream",
63+
"exposures",
64+
"column_reach",
65+
"column_reach_error",
66+
"exposures_error",
67+
"error",
68+
],
69+
}
70+
export const STRUCTURED_TOOLS: ReadonlySet<string> = new Set(Object.keys(STRUCTURED_FIELDS))
71+
const STRUCTURED_OUTPUT_SCHEMA = {
72+
type: "object" as const,
73+
properties: {
74+
report: { type: "string", description: "The tool's text report" },
75+
truncated: { type: "boolean", description: "True when the lineage fields were left out for size; the report is kept, shortened if it is too large on its own" },
76+
},
77+
required: ["report"],
78+
additionalProperties: true,
79+
}
80+
// The structured result reaches the model in Claude Code, so it stays within the same order of
81+
// size as a text result: past this, only the report is sent. Claude Code caps an MCP result at
82+
// 25k tokens by default and JSON of ids and edges runs ~3-4 characters a token, so this stays under that.
83+
const STRUCTURED_MAX_BYTES = 60_000
84+
85+
export function structuredResult(tool: string, metadata: Record<string, unknown>, report: string): Record<string, unknown> {
86+
// A tool that builds its own structured result (dbt_lineage from the dbt integration) hands it
87+
// over whole; otherwise the lineage fields of its metadata are passed.
88+
const own = metadata.structuredContent
89+
if (own && typeof own === "object" && !Array.isArray(own)) {
90+
// The text repeats every edge that `columnLineage` already carries: only its header goes with them.
91+
const head = report.split(/\n(?:Column edges \(|Trace of )/)[0]!.trim()
92+
const full = { ...(own as Record<string, unknown>), report: head }
93+
return Buffer.byteLength(JSON.stringify(full)) <= STRUCTURED_MAX_BYTES ? full : capReport(report, true)
94+
}
95+
const fields = Object.fromEntries((STRUCTURED_FIELDS[tool] ?? []).filter((key) => key in metadata).map((key) => [key, metadata[key]]))
96+
// dbt_lineage's text prints the same mappings and edges as `columns` and `edges`: keep its header.
97+
const head = tool === "dbt_lineage" && "columns" in fields ? report.split(/\n(?:Column Mappings|Lineage Edges):/)[0]!.trim() : report
98+
const full = { ...fields, report: head }
99+
return Buffer.byteLength(JSON.stringify(full)) <= STRUCTURED_MAX_BYTES ? full : capReport(report, true)
100+
}
101+
/** An error report held to the same cap as a result, so a huge exception message cannot exceed it. */
102+
export function structuredError(message: string): Record<string, unknown> {
103+
return capReport(message, false)
104+
}
105+
// `dropped`: the result's other fields were left out, so it is marked truncated even when the
106+
// report itself fits.
107+
function capReport(message: string, dropped: boolean): Record<string, unknown> {
108+
// Measured as sent: JSON escaping can grow the text (a newline is two bytes, a control
109+
// character six), so the cap applies to the serialized result, shortening until it fits.
110+
const size = (value: Record<string, unknown>) => Buffer.byteLength(JSON.stringify(value))
111+
const whole = dropped ? { report: message, truncated: true } : { report: message }
112+
if (size(whole) <= STRUCTURED_MAX_BYTES) return whole
113+
let report = message
114+
while (report.length > 0 && size({ report, truncated: true }) > STRUCTURED_MAX_BYTES) {
115+
report = report.slice(0, Math.floor(report.length * 0.9))
116+
}
117+
return { report, truncated: true }
118+
}
119+
// altimate_change end
120+
46121
/** Spilled results: a resource_link next to the text, which keeps the preview, row count and path. */
47122
function resultLinks(metadata: Record<string, unknown>) {
48123
// A batched sql_execute call reports one saved file per spilled query.
@@ -97,6 +172,7 @@ export async function createServer(options: ServeOptions = {}) {
97172
name,
98173
description: descriptions[name] ?? tool.description,
99174
inputSchema: tool.jsonSchema as McpTool["inputSchema"],
175+
...(STRUCTURED_TOOLS.has(name) && { outputSchema: STRUCTURED_OUTPUT_SCHEMA }),
100176
})), ...(workspace?.tools ?? [])],
101177
}))
102178
// Ask the user once per session when the model weighed two readings and the client can ask; any
@@ -146,19 +222,19 @@ export async function createServer(options: ServeOptions = {}) {
146222
const clarified = request.params.name === "answer_commit" ? await clarifyReading(request.params.arguments, result, extra.signal) : undefined
147223
if (clarified) return { content: [{ type: "text" as const, text: clarified }], isError: true }
148224
const isError = resultFailed(result)
149-
// A tool that builds a structured result (dbt_lineage) hands it over beside its text.
150-
const structured = result.metadata?.structuredContent
151225
return {
152226
content: [{ type: "text" as const, text: result.output }, ...resultLinks(result.metadata)],
153-
...(structured && typeof structured === "object" && !isError
154-
? { structuredContent: structured as Record<string, unknown> }
155-
: {}),
156227
isError,
228+
...(STRUCTURED_TOOLS.has(request.params.name) && {
229+
structuredContent: structuredResult(request.params.name, result.metadata, result.output),
230+
}),
157231
}
158232
} catch (error) {
233+
const message = error instanceof Error ? error.message : String(error)
159234
return {
160-
content: [{ type: "text" as const, text: error instanceof Error ? error.message : String(error) }],
235+
content: [{ type: "text" as const, text: message }],
161236
isError: true,
237+
...(STRUCTURED_TOOLS.has(request.params.name) && { structuredContent: structuredError(message) }),
162238
}
163239
}
164240
}

0 commit comments

Comments
 (0)