Repository navigation
Commit e0dc3da
feat: structured lineage and column reach for dbt_lineage and impact_analysis (#1635)
* feat: structured lineage and column reach for dbt_lineage and impact_analysis
Builds on the dbt integration's structured dbt_lineage result.
- dbt_lineage takes a project-relative SQL path or a snapshot as well as
a model, and with include_graph returns the model's graph from the
manifest (two levels up, everything downstream with exposures and
tests, capped at 500 nodes). The manifest fallback reports upstream
ids, column mappings and edges as data.
- impact_analysis with a column traces each downstream model's and
snapshot's compiled SQL in dependency order and reports column_reach:
certain readers with the columns they carry, possible readers
(filters and joins, select * over undocumented columns, shared table
names, ephemeral parents), untraced models and the exposures on each.
It matches models by unique_id and honours the dialect override.
- The MCP server declares an output schema for both tools. Structured
content is the tool's own result when it builds one (the dbt
integration's model lineage), else its lineage fields, always with the
text report and capped at 100 KB, errors included.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: dbt_lineage uses the adapter only for the named manifest's own project
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: an oversized structured result keeps its report within the cap and stays marked truncated
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs: the truncated flag says the report may be shortened too
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* feat: dbt_lineage include_graph returns the model's whole upstream, capped at 500 nodes and marked when cut
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: include_graph leaves out the tests of upstream models
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: address review comments on lineage structured results
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a column spelling that matches two columns differing only by case leaves their readers uncertain
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: an unsettled column spelling leaves the model's own dashboards uncertain, and project containment compares whole path segments
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: uncertainty for models the engine cannot fully trace, a token-safe structured cap, header-only adapter report, dashboards through metrics
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a model with no column edges is uncertain only when its SQL names the column
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test: a model below a partly traced reader is a possible reader
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a structured result passed through for a tool outside the declared set is held to the same cap
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* refactor: the column-name scan runs once per model; test rules out a certain reader for a constant model
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* style: indent the passthrough condition as Prettier does
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* refactor: drop the structured passthrough no tool reaches; dbt_lineage takes the declared, capped path
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: an uncompiled project's own manifest path keeps the dbt integration's answer, and whole-row reads are possible reads
- a manifest_path not written yet is placed by its path (nearest existing folder's links resolved): inside the adapter's project it keeps the adapter's lineage; elsewhere it is still another project's
- a parent's row used as a value (to_json(o), cast(o as varchar)) or a NATURAL JOIN makes a model a possible reader, never a confident non-reader
- downstream entries carry unique_id; a SQL path matches with ./ or backslashes
- count_big(*)/count_if(*) are not a select *; a backtick output alias is not a read
- include_graph's description says the graph is left out of MCP structured results
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a whole row is a row source passed whole to a call, so import CTEs stay confident and quoted, CTE and subquery rows are caught
- row sources: FROM/JOIN relations and aliases (quoted or not, never a keyword), CTE names and subquery aliases
- a whole row read beside the column passes the doubt on to the model's children
- a missing manifest with no dbt project above it names no other project; a link to a missing manifest cannot be placed
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: column reach flags a NATURAL JOIN only, and a manifest link is placed where it points
A pattern cannot tell a row source from a column (sum(x) as x, extract(year from d)), so whole-row
values are left to the lineage engine; a NATURAL JOIN is one keyword with no such ambiguity. A link
to a manifest not written yet is followed, and a path through a file is placed instead of throwing.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: every NATURAL join form counts and passes its doubt on, quoted names never do, and a relative link climbs from its real folder
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a NATURAL JOIN inside array brackets or after an escaped quote still counts
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a NATURAL JOIN is found in the raw SQL, so no dialect's quoting can hide one
Masking strings and quoted names to avoid a false positive let quoting rules that differ by dialect
hide real joins. The words inside a string or comment now count too, which only adds doubt.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: comments between NATURAL and JOIN are read as gaps, with no pass over the whole SQL
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: NATURAL JOIN lookup reads // and CR-ended comments, and stays bounded on any input
Each natural word is checked in a 1000-character window, and past 100 of them the SQL counts as
having one, which only adds doubt.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: Snowflake's DIRECTED NATURAL JOIN counts too
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a manifest is placed where the system opens it, through Filesystem.realLocation; an empty path is no model's; NATURAL JOIN gaps up to 10,000 characters
- links resolve before `..`, so `<project>/link/../x` lands where the link points; the duplicate resolver is gone, and a link loop cannot be placed
- a request that is empty once normalized never matches a model without a file path
- the window per natural grows to 10,000 characters, still bounded with the 100-word cap
- tests for count_big(*)/count_if(*), a backticked output alias, link/.. and the empty path
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a manifest is placed as the manifest reader opens it, and only link hops count toward the loop limit
loadRawManifest opens a path through realpathSync, which collapses `..` before following links, so
the project check does the same: the adapter's lineage and the graph read beside it always come from
one project. Climbing to the nearest existing folder no longer counts toward the 64-link limit.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* refactor: the requested SQL path is normalized once, and the link/.. test proves which manifest the graph reads
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* style: Prettier on the files this change left unformatted
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a column named through Snowflake IDENTIFIER() is a possible read, and a column reach that stopped early reports lower bounds
- IDENTIFIER('col') with a traced column, or a bind variable, makes the model uncertain and passes the doubt on, as a NATURAL JOIN does
- the text report says "at least" when column reach stopped early, and the exposures past the stop are unchecked, not non-readers
- the model parameter says a snapshot is found by name too
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: comments before or inside IDENTIFIER( are read as gaps, and its lookup is bounded like the NATURAL JOIN one
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: any mention of a traced column inside IDENTIFIER('…') counts, so a quoted name with a dot is never split
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a doubled quote inside an IDENTIFIER() name is read as one quote
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: an IDENTIFIER() string that is not a plain name counts as naming the column, so no escape can hide one
Escapes (\x72), backslash-escaped quotes and other characters are not decoded; such a string marks
the model uncertain, which only adds doubt.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: only an IDENTIFIER() whose whole argument is one plain string naming another column is cleared
Concatenations, bind and session variables, functions and unterminated strings all count as possibly
naming the column, so no argument form can produce a confident zero.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: a column reach that stopped early bounds every count, and the other exposures are not all called unchecked
- uncertain and untraced counts say "at least" too when column reach stopped early
- the other exposures' header says some of them were not checked, after the count
- the parents' traced columns are built once
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs: the stopped-early exposures note says some may not have been checked
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent a6412db commit e0dc3da
13 files changed
Lines changed: 2259 additions & 84 deletions
File tree
- packages/opencode
- src/altimate
- mcp-server
- native
- dbt
- tools
- test/altimate
- mcp-server
- tools
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
46 | 121 | | |
47 | 122 | | |
48 | 123 | | |
| |||
97 | 172 | | |
98 | 173 | | |
99 | 174 | | |
| 175 | + | |
100 | 176 | | |
101 | 177 | | |
102 | 178 | | |
| |||
146 | 222 | | |
147 | 223 | | |
148 | 224 | | |
149 | | - | |
150 | | - | |
151 | 225 | | |
152 | 226 | | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | 227 | | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
157 | 231 | | |
158 | 232 | | |
| 233 | + | |
159 | 234 | | |
160 | | - | |
| 235 | + | |
161 | 236 | | |
| 237 | + | |
162 | 238 | | |
163 | 239 | | |
164 | 240 | | |
| |||
0 commit comments