Skip to content

Commit 3924009

Browse files
anandgupta42claude
andcommitted
feat(dbt-tools): auto-run schema-verify after build --model
The agent has been ignoring skill-level instructions to call schema-verify before declaring done (v4 and v5 trace inspection showed the agent reads the rule, writes the intention in chain-of-thought, then doesn't run the command). Building a full harness-level validator framework that intercepts session termination is real engineering; this commit ships the smallest forcing mechanism that doesn't require touching the session loop: auto-trigger schema-verify inside the build command's own response. The agent now cannot see a successful `altimate-dbt build --model X` without also seeing the schema-verify verdict in the same tool result. The diff is in the JSON response under `schema_verify`, in-context where the agent's attention sits — much harder to ignore than a system-prompt skill rule. Behavior: - `build` without `--model` is unchanged (project-wide build, no per-model verify makes sense). - `build --model X` runs schema-verify on X after a successful build. The full structured result lives at `schema_verify`. - A verify failure does NOT mask the build's stdout — both are reported. Build status remains the success/error signal. - If verify itself errors (missing manifest, unbuilt table), the error is reported under `schema_verify.error` with a fix hint. Tests: - Updated existing build-test mocks to include parseManifest + getColumnsOfModel (no behavior change, just shape consistency). - New assertion: build --model X result now contains schema_verify. Skill: dbt-schema-verify body adds a note that the agent gets schema-verify "for free" inside build's response, so it doesn't need to call it twice for verification. This is a stepping stone — the full validator framework that intercepts session termination is the next iteration. This commit tests whether putting the diff inline with the build response is enough to break through the ignore-the-rule pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 519b817 commit 3924009

3 files changed

Lines changed: 47 additions & 3 deletions

File tree

‎.opencode/skills/dbt-schema-verify/SKILL.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,13 @@ just the last one. A `build` is not a verify.
4646
altimate-dbt schema-verify --model <name>
4747
```
4848

49+
**Note**: `altimate-dbt build --model <name>` already runs schema-verify
50+
automatically after a successful build and includes the verdict in its
51+
response under a `schema_verify` field. You will see the diff in the same
52+
result that reported the build outcome — read it there before deciding
53+
the task is done. If you need to re-check after editing, call
54+
`schema-verify` directly.
55+
4956
Returns a structured JSON result:
5057

5158
```json

‎packages/dbt-tools/src/commands/build.ts‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import type { DBTProjectIntegrationAdapter, CommandProcessResult } from "@altimateai/dbt-integration"
2+
import { schemaVerify } from "./schema-verify"
23

34
export async function build(adapter: DBTProjectIntegrationAdapter, args: string[]) {
45
const model = flag(args, "model")
@@ -12,7 +13,31 @@ export async function build(adapter: DBTProjectIntegrationAdapter, args: string[
1213
modelName: model,
1314
plusOperatorRight: downstream ? "+" : "",
1415
})
15-
return format(result)
16+
const formatted = format(result)
17+
18+
// Auto-run schema-verify after a successful single-model build. Surfacing
19+
// the verdict in the same tool result the agent just received is the
20+
// closest a CLI command can get to harness-level enforcement: the agent
21+
// cannot see a green build without also seeing the schema-verify diff.
22+
// Failures here are non-fatal — verify is advisory feedback, not a build
23+
// step. `no-spec` is reported so the agent knows there was no spec to
24+
// grade against.
25+
if (!("error" in formatted)) {
26+
try {
27+
const verify = await schemaVerify(adapter, ["--model", model])
28+
return { ...formatted, schema_verify: verify }
29+
} catch (e) {
30+
// Don't let verify failures mask a successful build.
31+
return {
32+
...formatted,
33+
schema_verify: {
34+
error: `schema-verify failed: ${e instanceof Error ? e.message : String(e)}. Run \`altimate-dbt schema-verify --model ${model}\` manually to inspect.`,
35+
},
36+
}
37+
}
38+
}
39+
40+
return formatted
1641
}
1742

1843
export async function run(adapter: DBTProjectIntegrationAdapter, args: string[]) {

‎packages/dbt-tools/test/build.test.ts‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@ function makeAdapter(overrides: Partial<DBTProjectIntegrationAdapter> = {}): DBT
88
unsafeBuildProjectImmediately: mock(() => Promise.resolve({ stdout: "project built", stderr: "" })),
99
unsafeRunModelImmediately: mock(() => Promise.resolve({ stdout: "", stderr: "" })),
1010
unsafeRunModelTestImmediately: mock(() => Promise.resolve({ stdout: "", stderr: "" })),
11+
// Auto-trigger after `build --model X` calls schema-verify too. Mock its
12+
// dependencies so the test exercises the build path without erroring on
13+
// missing adapter methods.
14+
parseManifest: mock(() => Promise.resolve({
15+
nodeMetaMap: { lookupByBaseName: mock(() => undefined), lookupByUniqueId: mock(() => undefined), nodes: mock(() => []) },
16+
})),
17+
getColumnsOfModel: mock(() => Promise.resolve([])),
1118
dispose: mock(() => Promise.resolve()),
1219
...overrides,
1320
} as unknown as DBTProjectIntegrationAdapter
@@ -22,7 +29,7 @@ describe("build command", () => {
2229
expect(result).toEqual({ stdout: "project built" })
2330
})
2431

25-
test("build --model <name> builds single model", async () => {
32+
test("build --model <name> builds single model and auto-runs schema-verify", async () => {
2633
const adapter = makeAdapter()
2734
const result = await build(adapter, ["--model", "orders"])
2835
expect(adapter.unsafeBuildModelImmediately).toHaveBeenCalledTimes(1)
@@ -32,7 +39,12 @@ describe("build command", () => {
3239
plusOperatorRight: "",
3340
})
3441
expect(adapter.unsafeBuildProjectImmediately).not.toHaveBeenCalled()
35-
expect(result).toEqual({ stdout: "model built" })
42+
// After a successful single-model build, schema-verify is auto-run and
43+
// its result appears under `schema_verify`. The agent cannot see a green
44+
// build without also seeing the shape diff.
45+
expect(adapter.parseManifest).toHaveBeenCalledTimes(1)
46+
expect((result as Record<string, unknown>).stdout).toBe("model built")
47+
expect((result as Record<string, unknown>).schema_verify).toBeDefined()
3648
})
3749

3850
test("build --model <name> --downstream sets plusOperatorRight", async () => {

0 commit comments

Comments
 (0)