Skip to content

Commit 0724de3

Browse files
anandgupta42claude
andcommitted
fix: [#849] address code-review findings in validator framework
All issues identified in the consensus review (Claude + 8 models) are addressed in this commit. **MAJOR fixes (blocking merge)** - Add subprocess timeout (`ALTIMATE_VALIDATORS_TIMEOUT_MS`, default 60 s) to `runDbtTest` and `runSchemaVerify` — prevents the agent loop from hanging indefinitely on stalled DuckDB connections or warehouse I/O. Kills the child process with SIGKILL on timeout. - Fix shadow telemetry gate: move `ValidatorRegistry.runAll()` and the per-validator `Telemetry.track()` loop outside the `validatorsEnabled` check in `prompt.ts`. Telemetry now fires regardless of the feature flag, fulfilling the "measure before enforce" promise stated in the inline comment. - Fix nested dbt project cwd bug: replace `isDbtProject(cwd): boolean` with `findDbtProjectRoot(cwd): Promise<string | null>` which returns the directory that actually contains `dbt_project.yml`. Both validators now pass that root as `cwd` to subprocess invocations and as the base for `modelsModifiedSince`, preventing the "not a dbt project" error when the project is one level below the working dir. - Extract shared helpers into `validator-utils.ts`: `findDbtProjectRoot`, `modelsModifiedSince`, `modelNameFromPath`, `extractLastJsonObject`. Both validator files now import from the shared module. The validated `extractLastJsonObject` rejects stray JSON fragments (checks for `verdict`/`error`/`model`/`stdout`/`columns_*` keys) — the laxer version that was only in `dbt-tests-pass.ts` is gone. - Add tests: 39 unit tests covering `extractLastJsonObject` (8 cases), `modelNameFromPath`, `findDbtProjectRoot` (5 cases), `modelsModifiedSince` (7 cases), and `parseDbtTestOutput` (10 cases including dbt 1.x format, ANSI prefixes, NO-OP variant, duplicate names, `[FAIL`/`[ERROR` token exclusion). **MINOR fixes** - Track spawn failures separately in `dbt-tests-pass.check()` and `dbt-schema-verify.check()`. `details.spawn_failures` now appears in the validator result so operators can distinguish "skipped model" from "passed model". - Add retries-exhausted telemetry: when `validatorRetryCount >= maxValidatorRetries` with failures outstanding, emit `validator_retries_exhausted` event and a `log.warn` so the session doesn't silently appear as "completed" in the operator dashboard. - Parallel model checking: both `check()` functions now use `Promise.all` instead of a sequential `for` loop. **NITS** - Named regex groups in `parseDbtTestOutput` — replaces positional `summaryMatch[1]` / `[3]` / `[5]` captures; resilient to dbt reordering summary fields. - Path separator: `modelsModifiedSince` and `modelNameFromPath` now use `path.sep` / `path.basename` instead of hardcoded `"/"`. - Fix `ls | head -1` in `build-local-tarball.sh`: derive exact tarball name from `VERSION` variable; error out explicitly if not present. - Fix stale comment "Limited to two-level deep search" in `dbt-schema-verify.ts` (actual depth was 4; comment is removed). Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
1 parent 7ca5a36 commit 0724de3

7 files changed

Lines changed: 703 additions & 287 deletions

File tree

‎benchmark/ade-bench/altimate_code_agent/build-local-tarball.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,9 @@ EOF
8181
# Pack with bun pm pack — produces altimate-code-<ver>-local.tgz
8282
( cd "$STAGE" && bun pm pack >/dev/null )
8383

84-
TARBALL="$(ls -1 "$STAGE"/altimate-code-*.tgz | head -1)"
85-
if [[ -z "$TARBALL" ]]; then
86-
echo "pack failed: no tarball produced" >&2
84+
TARBALL="$STAGE/altimate-code-${VERSION}-local.tgz"
85+
if [[ ! -f "$TARBALL" ]]; then
86+
echo "pack failed: expected $TARBALL" >&2
8787
exit 1
8888
fi
8989
mv "$TARBALL" "$SCRIPT_DIR/altimate-code-local.tgz"

‎packages/opencode/src/altimate/validators/dbt-schema-verify.ts‎

Lines changed: 45 additions & 143 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,15 @@
1414
* by the agent — see types.ts header for the rationale.
1515
*/
1616

17-
import { promises as fs } from "fs"
18-
import { join } from "path"
1917
import { spawn } from "child_process"
2018
import type { Validator, ValidatorContext, ValidatorResult } from "../../session/validators/types"
19+
import {
20+
VALIDATOR_TIMEOUT_MS,
21+
findDbtProjectRoot,
22+
modelsModifiedSince,
23+
modelNameFromPath,
24+
extractLastJsonObject,
25+
} from "./validator-utils"
2126

2227
interface SchemaVerifyOutput {
2328
model?: string
@@ -29,150 +34,27 @@ interface SchemaVerifyOutput {
2934
error?: string
3035
}
3136

32-
/**
33-
* Best-effort check that the working directory looks like a dbt project.
34-
* Scans the directory itself and one level of subdirs for `dbt_project.yml`.
35-
*/
36-
async function isDbtProject(cwd: string): Promise<boolean> {
37-
try {
38-
const direct = await fs.stat(join(cwd, "dbt_project.yml")).then(
39-
() => true,
40-
() => false,
41-
)
42-
if (direct) return true
43-
// Some benchmark layouts nest the project one level deep. Cheap scan.
44-
const entries = await fs.readdir(cwd, { withFileTypes: true })
45-
for (const e of entries) {
46-
if (!e.isDirectory()) continue
47-
const nested = await fs.stat(join(cwd, e.name, "dbt_project.yml")).then(
48-
() => true,
49-
() => false,
50-
)
51-
if (nested) return true
52-
}
53-
return false
54-
} catch {
55-
return false
56-
}
57-
}
58-
59-
/**
60-
* Find dbt model `.sql` files under the working directory that were modified
61-
* since the session started. Limited to two-level deep search to keep cost
62-
* bounded on large projects.
63-
*/
64-
async function modelsModifiedSince(cwd: string, sinceMs: number): Promise<string[]> {
65-
const found: string[] = []
66-
async function scan(dir: string, depth: number): Promise<void> {
67-
if (depth > 4) return
68-
let entries: import("fs").Dirent[]
69-
try {
70-
entries = await fs.readdir(dir, { withFileTypes: true })
71-
} catch {
72-
return
73-
}
74-
for (const entry of entries) {
75-
if (entry.name.startsWith(".") || entry.name === "node_modules" || entry.name === "target") continue
76-
const full = join(dir, entry.name)
77-
if (entry.isDirectory()) {
78-
await scan(full, depth + 1)
79-
} else if (entry.isFile() && entry.name.endsWith(".sql")) {
80-
try {
81-
const stat = await fs.stat(full)
82-
if (stat.mtimeMs >= sinceMs) {
83-
// Convention: dbt models live under a `models/` ancestor.
84-
if (full.split("/").includes("models")) {
85-
found.push(full)
86-
}
87-
}
88-
} catch {
89-
// ignore unstattable files
90-
}
91-
}
92-
}
93-
}
94-
await scan(cwd, 0)
95-
return found
96-
}
97-
98-
/** Extract bare model name from a `.sql` file path. `models/marts/foo.sql` -> `foo`. */
99-
function modelNameFromPath(p: string): string {
100-
const base = p.split("/").pop() ?? p
101-
return base.replace(/\.sql$/i, "")
102-
}
103-
10437
/**
10538
* Extract a SchemaVerifyOutput JSON object from mixed stdout.
10639
* `altimate-dbt schema-verify` may emit dbt log noise (ANSI codes, parser
107-
* warnings) before the verdict JSON. Strategy:
108-
* 1. Try JSON.parse on the full stdout (fast path for clean output).
109-
* 2. Otherwise, scan for the LAST balanced `{...}` substring and parse that.
110-
*
111-
* Returns null if no parseable JSON object is found.
40+
* warnings) before the verdict JSON. Delegates to the shared
41+
* extractLastJsonObject utility which already handles noisy stdout and
42+
* validates the envelope shape.
11243
*/
11344
function parseSchemaVerifyOutput(stdout: string): SchemaVerifyOutput | null {
114-
if (!stdout) return null
115-
// Fast path: stdout is pure JSON
116-
try {
117-
return JSON.parse(stdout) as SchemaVerifyOutput
118-
} catch {
119-
// fall through
120-
}
121-
// Find each `{` and try to parse a JSON object starting there. Take the
122-
// LAST one that parses to a SchemaVerifyOutput-shaped result. dbt log
123-
// noise may include `{` inside log lines, so we accept the last verdict
124-
// (verdict / model / error key) we can parse end-to-end.
125-
let best: SchemaVerifyOutput | null = null
126-
for (let i = 0; i < stdout.length; i++) {
127-
if (stdout[i] !== "{") continue
128-
// Scan forward to find the matching closing brace.
129-
let depth = 0
130-
let inString: '"' | null = null
131-
let escaped = false
132-
for (let j = i; j < stdout.length; j++) {
133-
const ch = stdout[j]
134-
if (escaped) {
135-
escaped = false
136-
continue
137-
}
138-
if (ch === "\\") {
139-
escaped = true
140-
continue
141-
}
142-
if (inString) {
143-
if (ch === inString) inString = null
144-
continue
145-
}
146-
if (ch === '"') {
147-
inString = '"'
148-
continue
149-
}
150-
if (ch === "{") depth++
151-
else if (ch === "}") {
152-
depth--
153-
if (depth === 0) {
154-
try {
155-
const parsed = JSON.parse(stdout.slice(i, j + 1)) as SchemaVerifyOutput
156-
if (
157-
parsed &&
158-
(parsed.verdict !== undefined || parsed.error !== undefined || parsed.model !== undefined)
159-
) {
160-
best = parsed
161-
}
162-
} catch {
163-
// not parseable; skip
164-
}
165-
break
166-
}
167-
}
168-
}
169-
}
170-
return best
45+
const obj = extractLastJsonObject(stdout)
46+
if (!obj) return null
47+
return obj as SchemaVerifyOutput
17148
}
17249

17350
/**
17451
* Run `altimate-dbt schema-verify --model <name>` and parse its JSON output.
175-
* Returns null on spawn failure so the caller can fall back gracefully.
52+
*
53+
* Times out after ALTIMATE_VALIDATORS_TIMEOUT_MS (default 60 s) and kills the
54+
* subprocess to prevent the agent loop from hanging indefinitely on stalled
55+
* warehouse connections or DuckDB file-lock contention.
56+
*
57+
* Returns null on spawn failure so the caller can track it separately.
17658
*/
17759
async function runSchemaVerify(model: string, cwd: string): Promise<SchemaVerifyOutput | null> {
17860
const debug = process.env.ALTIMATE_VALIDATORS_DEBUG === "1"
@@ -182,11 +64,16 @@ async function runSchemaVerify(model: string, cwd: string): Promise<SchemaVerify
18264
env: process.env,
18365
stdio: ["ignore", "pipe", "pipe"],
18466
})
67+
const timer = setTimeout(() => {
68+
child.kill("SIGKILL")
69+
resolve({ error: `timed out after ${VALIDATOR_TIMEOUT_MS}ms` })
70+
}, VALIDATOR_TIMEOUT_MS)
18571
let stdout = ""
18672
let stderr = ""
18773
child.stdout.on("data", (chunk) => (stdout += String(chunk)))
18874
child.stderr.on("data", (chunk) => (stderr += String(chunk)))
18975
child.on("error", (e) => {
76+
clearTimeout(timer)
19077
if (debug) {
19178
// eslint-disable-next-line no-console
19279
console.error(
@@ -197,6 +84,7 @@ async function runSchemaVerify(model: string, cwd: string): Promise<SchemaVerify
19784
resolve(null)
19885
})
19986
child.on("close", (code) => {
87+
clearTimeout(timer)
20088
if (debug) {
20189
// eslint-disable-next-line no-console
20290
console.error(
@@ -255,21 +143,33 @@ export const DbtSchemaVerifyValidator: Validator = {
255143

256144
async appliesTo(ctx: ValidatorContext): Promise<boolean> {
257145
// Only run for sessions that took place inside a dbt project. Quick check.
258-
return isDbtProject(ctx.workingDirectory)
146+
return (await findDbtProjectRoot(ctx.workingDirectory)) !== null
259147
},
260148

261149
async check(ctx: ValidatorContext): Promise<ValidatorResult> {
262-
const touched = await modelsModifiedSince(ctx.workingDirectory, ctx.sessionStartMs)
150+
const dbtRoot = await findDbtProjectRoot(ctx.workingDirectory)
151+
if (!dbtRoot) return { ok: true, details: { models_touched: 0 } }
152+
153+
const touched = await modelsModifiedSince(dbtRoot, ctx.sessionStartMs)
263154
if (touched.length === 0) {
264155
// No models touched — nothing to verify.
265156
return { ok: true, details: { models_touched: 0 } }
266157
}
267158

159+
// Run all schema-verify calls in parallel; track spawn failures separately.
160+
let spawnFailures = 0
161+
const outputs = await Promise.all(
162+
touched.map((path) => runSchemaVerify(modelNameFromPath(path), dbtRoot)),
163+
)
268164
const results: SchemaVerifyOutput[] = []
269-
for (const path of touched) {
270-
const name = modelNameFromPath(path)
271-
const out = await runSchemaVerify(name, ctx.workingDirectory)
272-
if (out) results.push({ ...out, model: out.model ?? name })
165+
for (let i = 0; i < outputs.length; i++) {
166+
const out = outputs[i]!
167+
const name = modelNameFromPath(touched[i]!)
168+
if (out !== null) {
169+
results.push({ ...out, model: out.model ?? name })
170+
} else {
171+
spawnFailures++
172+
}
273173
}
274174

275175
const mismatches = results.filter((r) => r.verdict === "mismatch")
@@ -286,6 +186,7 @@ export const DbtSchemaVerifyValidator: Validator = {
286186
match: matches,
287187
no_spec: noSpec,
288188
errored,
189+
spawn_failures: spawnFailures,
289190
},
290191
}
291192
}
@@ -303,6 +204,7 @@ export const DbtSchemaVerifyValidator: Validator = {
303204
mismatch: mismatches.length,
304205
no_spec: noSpec,
305206
errored,
207+
spawn_failures: spawnFailures,
306208
mismatch_models: mismatches.map((m) => m.model).filter(Boolean),
307209
},
308210
}

0 commit comments

Comments
 (0)