Skip to content

Commit 81b6df2

Browse files
anandgupta42claude
andcommitted
fix: [#849] address PR review comments — validator hardening + agent fixes
Addresses all unresolved review threads from coderabbitai and cubic-dev-ai on PR #849 (feat/validator-framework). **Critical fixes** - `build-local-tarball.sh`: REPO_ROOT traversal depth 6→3 `..` segments (script is at `benchmark/ade-bench/altimate_code_agent/`, 3 levels deep) - `prompt.ts`: explicit `continue` after `validatorRetryCount++` to make the retry loop intent unambiguous (was falling through to bottom-of-loop `continue` correctly but implicitly) - `prompt.ts`: `workingDirectory` now uses `Instance.directory` instead of `process.cwd()` to match the session's actual working directory **Major fixes** - `altimate_code_agent.py`: `shlex.quote()` on `self._model_name` before shell interpolation to prevent injection via model name strings - `altimate-code-setup.sh`: `@latest` fallback replaced with `exit 1` for benchmark reproducibility; config file now written with `chmod 600` - `dbt-schema-verify.ts`: fail closed on errors — spawn failures now pushed into `results` as error entries; `ok` check requires `errored === 0` so unverifiable models don't silently pass the completion gate - `system.ts`: XML-escape `skill.name` before embedding in `<auto_loaded_skill name="...">` attribute via `escapeXmlAttr()` - `system.ts`: remove `.catch(() => [])` inside `anyMatchInWorktree` so errors propagate to the outer try/catch; `autoLoadLog.warn(...)` is now reachable - `registry.ts`: `appliesTo()` exceptions now surface error details as a soft-pass result entry instead of being silently swallowed **Minor fixes — validator-utils.ts hardening** - `VALIDATOR_TIMEOUT_MS`: finite/positive guard against NaN, 0, or negative env-var values (all fall back to the 60 s default) - `modelsModifiedSince`: case-insensitive `.sql` check (`.toLowerCase()`) for consistency with `modelNameFromPath` which already uses `/\.sql$/i` - New `runWithConcurrencyLimit` helper (max 4 concurrent by default, env override via `ALTIMATE_VALIDATORS_CONCURRENCY`) replaces unbounded `Promise.all` in both `dbt-tests-pass.ts` and `dbt-schema-verify.ts` **Docs / research corrections** - `findings.md`: timing table row renamed "Step-to-step intervals (start-to-start)" to clarify it includes step duration, not just gaps - `findings.md`: f1011 description corrected — `check_option_b` FAILED (per Appendix C), not passed - `findings.md`: per-domain failure counts corrected (asana 4→3, f1 5→4; total now sums to 19, consistent with headline 78-59=19) - `dbt-unit-tests/SKILL.md`: removed untestable "empty partition" window function guidance; replaced with partition-of-1 and tie-break row cases - `dbt-schema-verify/SKILL.md`: fallback verification glob broadened from `schema.yml` to `**/*.yml` to catch `_models.yml` and other conventions Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
1 parent 8a33919 commit 81b6df2

13 files changed

Lines changed: 116 additions & 29 deletions

File tree

‎.opencode/skills/dbt-schema-verify/SKILL.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,10 @@ Then run `altimate-dbt build --model <name>` again, then re-run
114114
If `which altimate-dbt` returns nothing, do the same diff by hand:
115115

116116
```bash
117-
# 1. Read expected columns from schema.yml
118-
cat models/**/schema.yml | grep -A 50 "name: <name>" # or yq
117+
# 1. Read expected columns from any YAML spec under models/
118+
# dbt allows any .yml filename; common patterns include schema.yml,
119+
# _models.yml, models.yml, sources.yml, etc.
120+
cat models/**/*.yml | grep -A 50 "name: <name>" # or: yq eval '...' models/**/*.yml
119121

120122
# 2. Read actual columns from the materialized table
121123
dbt show --select <name> --limit 0

‎.opencode/skills/dbt-unit-tests/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ description: Generate dbt unit tests automatically for any model. Analyzes SQL l
4040
- **`/` division** → row where the denominator is `0` or `NULL`
4141
- **`CASE WHEN`** → at least one row matching each branch, including the implicit `ELSE NULL` if no explicit `ELSE` is set
4242
- **`COALESCE` / `IFNULL`** → row where every argument is `NULL`
43-
- **Window functions (`OVER`)** → an empty partition, a partition of size 1, and a row at the partition boundary
43+
- **Window functions (`OVER`)** → a partition of size 1 (single-row group exercises rank/first/last edge cases), a row at the partition boundary, and a tie-break row (two rows with the same ORDER BY key)
4444
- **Date arithmetic / date spines** → a row at the start of range, end of range, and a gap day with no events
4545
- **Aggregations with `GROUP BY`** → at least one group of size 1 (often masks fan-out bugs) and one group whose key is `NULL`
4646
- **Incremental merge keys** → both an "insert" row and an "update" row matching an existing key
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Build artifacts produced by build-local-tarball.sh
2+
altimate-code-local.tgz
3+
.stage/

‎benchmark/ade-bench/altimate_code_agent/altimate-code-setup.sh‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,11 @@ if [[ -f "$LOCAL_TARBALL" ]]; then
2525
cp "$SRC" "$PKG_BIN_DIR/.altimate"
2626
chmod 755 "$PKG_BIN_DIR/.altimate-code" "$PKG_BIN_DIR/.altimate"
2727
else
28-
echo "Local tarball not staged; falling back to latest published"
29-
npm install -g --no-audit --no-fund @altimateai/altimate-code@latest
28+
# Abort rather than silently install an uncontrolled @latest version.
29+
# Benchmark reproducibility requires the exact local build under test.
30+
# Stage the tarball via build-local-tarball.sh before running this script.
31+
echo "ERROR: local tarball $LOCAL_TARBALL not found. Run build-local-tarball.sh first." >&2
32+
exit 1
3033
fi
3134

3235
altimate-code --version
@@ -97,6 +100,7 @@ ${PROVIDERS}
97100
}
98101
}
99102
EOF
103+
chmod 600 "$CONFIG_DIR/altimate-code.json"
100104
echo "Wrote altimate-code config; providers registered:"
101105
grep -oE '"(azure-foundry|openrouter)":' "$CONFIG_DIR/altimate-code.json" | tr -d '":' | sed 's/^/ - /'
102106
else

‎benchmark/ade-bench/altimate_code_agent/altimate_code_agent.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ def _run_agent_commands(self, task_prompt: str) -> list[TerminalCommand]:
225225
command = f"echo 'AGENT RESPONSE: ' && altimate-code run --format json --yolo"
226226

227227
if self._model_name:
228-
command += f" --model {self._model_name}"
228+
command += f" --model {shlex.quote(self._model_name)}"
229229
command += f" --max-turns 80 {escaped_prompt}"
230230

231231
return [

‎benchmark/ade-bench/altimate_code_agent/build-local-tarball.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
set -euo pipefail
99

1010
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
11-
REPO_ROOT="$(cd "$SCRIPT_DIR/../../../../../.." && pwd)"
11+
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
1212
PKG_DIR="$REPO_ROOT/packages/opencode"
1313
DBT_TOOLS_DIR="$REPO_ROOT/packages/dbt-tools"
1414
SKILLS_DIR="$REPO_ROOT/.opencode/skills"

‎packages/opencode/src/altimate/validators/dbt-schema-verify.ts‎

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,12 @@ import { spawn } from "child_process"
1818
import type { Validator, ValidatorContext, ValidatorResult } from "../../session/validators/types"
1919
import {
2020
VALIDATOR_TIMEOUT_MS,
21+
VALIDATOR_CONCURRENCY,
2122
findDbtProjectRoot,
2223
modelsModifiedSince,
2324
modelNameFromPath,
2425
extractLastJsonObject,
26+
runWithConcurrencyLimit,
2527
} from "./validator-utils"
2628

2729
interface SchemaVerifyOutput {
@@ -156,10 +158,13 @@ export const DbtSchemaVerifyValidator: Validator = {
156158
return { ok: true, details: { models_touched: 0 } }
157159
}
158160

159-
// Run all schema-verify calls in parallel; track spawn failures separately.
161+
// Run schema-verify calls with a bounded concurrency limit to prevent
162+
// resource contention from too many simultaneous dbt processes.
160163
let spawnFailures = 0
161-
const outputs = await Promise.all(
162-
touched.map((path) => runSchemaVerify(modelNameFromPath(path), dbtRoot)),
164+
const outputs = await runWithConcurrencyLimit(
165+
touched,
166+
(path) => runSchemaVerify(modelNameFromPath(path), dbtRoot),
167+
VALIDATOR_CONCURRENCY,
163168
)
164169
const results: SchemaVerifyOutput[] = []
165170
for (let i = 0; i < outputs.length; i++) {
@@ -169,6 +174,9 @@ export const DbtSchemaVerifyValidator: Validator = {
169174
results.push({ ...out, model: out.model ?? name })
170175
} else {
171176
spawnFailures++
177+
// Track spawn failures as errored results so they appear in telemetry
178+
// and detail counts rather than being silently dropped (fails open).
179+
results.push({ model: name, error: "spawn failed: subprocess did not start" })
172180
}
173181
}
174182

@@ -177,7 +185,10 @@ export const DbtSchemaVerifyValidator: Validator = {
177185
const matches = results.filter((r) => r.verdict === "match").length
178186
const errored = results.filter((r) => r.error).length
179187

180-
if (mismatches.length === 0) {
188+
// Fail closed: return ok only when every model was verified and none mismatched.
189+
// Errors (spawn failures, schema-verify tool errors) prevent a clean pass because
190+
// we cannot rule out drift on models we failed to inspect.
191+
if (mismatches.length === 0 && errored === 0) {
181192
return {
182193
ok: true,
183194
details: {
@@ -191,12 +202,19 @@ export const DbtSchemaVerifyValidator: Validator = {
191202
}
192203
}
193204

205+
const reason =
206+
mismatches.length > 0
207+
? `${mismatches.length} of ${results.length} models you edited have a column-shape mismatch against schema.yml. The build may be green, but equality tests will fail.`
208+
: `${errored} model(s) could not be schema-verified (spawn or tool errors) — schema drift cannot be ruled out. Investigate before declaring done.`
209+
194210
return {
195211
ok: false,
196-
reason: `${mismatches.length} of ${results.length} models you edited have a column-shape mismatch against schema.yml. The build may be green, but equality tests will fail.`,
212+
reason,
197213
fixHint:
198-
formatFixHint(mismatches) +
199-
`\n\nFix the model SQL to match the schema.yml spec (do not edit the spec), rebuild, and the harness will re-check before declaring done.`,
214+
mismatches.length > 0
215+
? formatFixHint(mismatches) +
216+
`\n\nFix the model SQL to match the schema.yml spec (do not edit the spec), rebuild, and the harness will re-check before declaring done.`
217+
: `Run \`altimate-dbt schema-verify <model>\` manually to diagnose the error. Check that altimate-dbt is on PATH and that the dbt project compiles cleanly.`,
200218
details: {
201219
models_touched: touched.length,
202220
verified: results.length,

‎packages/opencode/src/altimate/validators/dbt-tests-pass.ts‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,12 @@ import { spawn } from "child_process"
2121
import type { Validator, ValidatorContext, ValidatorResult } from "../../session/validators/types"
2222
import {
2323
VALIDATOR_TIMEOUT_MS,
24+
VALIDATOR_CONCURRENCY,
2425
findDbtProjectRoot,
2526
modelsModifiedSince,
2627
modelNameFromPath,
2728
extractLastJsonObject,
29+
runWithConcurrencyLimit,
2830
} from "./validator-utils"
2931

3032
export interface TestSummary {
@@ -183,11 +185,13 @@ export const DbtTestsPassValidator: Validator = {
183185
return { ok: true, details: { models_touched: 0 } }
184186
}
185187

186-
// Run all model tests in parallel; track spawn failures separately so the
187-
// caller can see which models were not verifiable vs which passed/failed.
188+
// Run model tests with a bounded concurrency limit to prevent resource
189+
// contention from spawning too many simultaneous dbt processes (flaky failures).
188190
let spawnFailures = 0
189-
const outputs = await Promise.all(
190-
touched.map((path) => runDbtTest(modelNameFromPath(path), dbtRoot)),
191+
const outputs = await runWithConcurrencyLimit(
192+
touched,
193+
(path) => runDbtTest(modelNameFromPath(path), dbtRoot),
194+
VALIDATOR_CONCURRENCY,
191195
)
192196
const results: TestRunOutput[] = []
193197
for (const out of outputs) {

‎packages/opencode/src/altimate/validators/validator-utils.ts‎

Lines changed: 44 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,14 @@ import { join, sep, basename } from "path"
2121
* killing it and treating the model as unverifiable. Overrideable via
2222
* ALTIMATE_VALIDATORS_TIMEOUT_MS for benchmark environments where dbt startup
2323
* time varies.
24+
*
25+
* Parses with a finite/positive guard: NaN, 0, or negative values are rejected
26+
* and fall back to the 60 s default, preventing immediate SIGKILL of the process.
2427
*/
25-
export const VALIDATOR_TIMEOUT_MS = Number(process.env.ALTIMATE_VALIDATORS_TIMEOUT_MS ?? "60000")
28+
const DEFAULT_TIMEOUT_MS = 60_000
29+
const _parsed = Number(process.env.ALTIMATE_VALIDATORS_TIMEOUT_MS)
30+
export const VALIDATOR_TIMEOUT_MS =
31+
Number.isFinite(_parsed) && _parsed > 0 ? _parsed : DEFAULT_TIMEOUT_MS
2632

2733
// ---------------------------------------------------------------------------
2834
// Project detection
@@ -84,7 +90,7 @@ export async function modelsModifiedSince(cwd: string, sinceMs: number): Promise
8490
const full = join(dir, entry.name)
8591
if (entry.isDirectory()) {
8692
await scan(full, depth + 1)
87-
} else if (entry.isFile() && entry.name.endsWith(".sql")) {
93+
} else if (entry.isFile() && entry.name.toLowerCase().endsWith(".sql")) {
8894
try {
8995
const stat = await fs.stat(full)
9096
if (stat.mtimeMs >= sinceMs) {
@@ -116,6 +122,42 @@ export function modelNameFromPath(p: string): string {
116122
return basename(p).replace(/\.sql$/i, "")
117123
}
118124

125+
// ---------------------------------------------------------------------------
126+
// Concurrency utilities
127+
// ---------------------------------------------------------------------------
128+
129+
/**
130+
* Run `fn` over `items` with at most `limit` concurrent tasks at a time.
131+
*
132+
* Unbounded Promise.all over model lists can spawn too many simultaneous dbt
133+
* subprocesses, causing resource contention, port conflicts, or flaky results.
134+
* This helper caps the active workers while preserving output order.
135+
*/
136+
export async function runWithConcurrencyLimit<In, Out>(
137+
items: In[],
138+
fn: (item: In) => Promise<Out>,
139+
limit: number,
140+
): Promise<Out[]> {
141+
const results: Out[] = new Array(items.length)
142+
let next = 0
143+
async function worker(): Promise<void> {
144+
while (next < items.length) {
145+
const i = next++
146+
results[i] = await fn(items[i]!)
147+
}
148+
}
149+
const workers = Array.from({ length: Math.min(limit, items.length) }, worker)
150+
await Promise.all(workers)
151+
return results
152+
}
153+
154+
/** Maximum simultaneous altimate-dbt subprocesses per validator run. */
155+
export const VALIDATOR_CONCURRENCY =
156+
(() => {
157+
const v = Number(process.env.ALTIMATE_VALIDATORS_CONCURRENCY)
158+
return Number.isFinite(v) && v > 0 ? Math.floor(v) : 4
159+
})()
160+
119161
// ---------------------------------------------------------------------------
120162
// JSON extraction
121163
// ---------------------------------------------------------------------------

‎packages/opencode/src/session/prompt.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1116,7 +1116,7 @@ export namespace SessionPrompt {
11161116
try {
11171117
const vCtx = {
11181118
sessionID,
1119-
workingDirectory: process.cwd(),
1119+
workingDirectory: Instance.directory,
11201120
sessionStartMs: sessionStartTime,
11211121
step,
11221122
retryCount: validatorRetryCount,
@@ -1200,8 +1200,7 @@ export namespace SessionPrompt {
12001200
})
12011201

12021202
validatorRetryCount++
1203-
// Fall through to `continue`; the next iteration's top-of-loop
1204-
// sees the newer user message and does NOT break.
1203+
continue
12051204
} else if (failures.length > 0 && validatorsEnabled && validatorRetryCount >= maxValidatorRetries) {
12061205
// Retry budget exhausted with outstanding failures. Session will
12071206
// terminate on the natural break below. Emit an explicit signal so

0 commit comments

Comments
 (0)