Skip to content

Commit cf8c8d1

Browse files
mayoalexanderclaude
andcommitted
fix(eval): a score that cannot go down is not a score
`iris eval run` reported "Tests passed 7/7". It counted every non-empty reply as a pass: const passed = results.filter((r) => r.status === "completed").length // "completed" = summary.trim().length > 0 Four of the seven scenarios asserted NOTHING — introduction, creative, tool-awareness, summary — so they passed by existing. The two that did have keyword checks had their results printed and then excluded from the count. The number could not go down, which made it a liveness probe wearing a score's clothes. Two changes. 1. A test with no assertion cannot pass; it can only run. Verdicts are now pass | fail | unscored, and the summary reads "3 / 3 scored · 4 unscored (ran, asserted nothing)" instead of 7/7. Same agent, same run, honest number. 2. A second suite that tests THIS agent rather than the model underneath it. `core` (Eiffel Tower, 17*24, a haiku) passes identically whether the agent's tools, YAML registry and integrations work or are entirely broken, because none of it is exercised. `agent` asserts: tool-use the agent CALLED a tool — toolsUsed non-empty. The old "what tools do you have?" is a self-report the agent will happily give from its prompt header while every one of them is unreachable. grounding it reached the workspace at all no-fabrication it declines to invent when asked about something that does not exist Verified against PATTY (#420), not asserted: tool-use called Pathways_get_firm_directory, grounding called SearchKnowledgeBaseTool. And it fails honestly — agent #11 returned "Resource not found" and scored 0/3 rather than passing vacuously. no-fabrication is documented in-code as flaky BY CONSTRUCTION, because it is a lexical proxy for a semantic property: the same agent on the same prompt hedged in wording my list matched once and missed twice. The caveat says to read the reply on a failure rather than conclude fabrication, and not to put it in a threshold. Better to ship the limitation written down than a check that looks deterministic and is not. No MCP change. `iris_run` already executes any CLI command, so the MCP server stays at seven tools and picks this up for free. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JQ9915AheUUs2qL2D2u62K
1 parent 84b9411 commit cf8c8d1

1 file changed

Lines changed: 134 additions & 23 deletions

File tree

‎packages/opencode/src/cli/cmd/platform-eval.ts‎

Lines changed: 134 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -13,29 +13,106 @@ import { writeFileSync } from "fs"
1313

1414
interface EvalTest {
1515
name: string
16+
suite: "core" | "agent"
1617
prompt: string
18+
/** All of these must appear in the reply. */
1719
expectKeywords?: string[]
20+
/** At least ONE of these must appear — for checks with many valid phrasings. */
21+
expectAny?: string[]
22+
/** The agent must have actually CALLED a tool, not merely described having them. */
23+
expectTools?: boolean
1824
}
1925

26+
// Two suites, because they answer different questions and only one of them is
27+
// about IRIS.
28+
//
29+
// `core` asks "is the model alive and following instructions" — Eiffel Tower,
30+
// 17*24, a haiku. Useful as a smoke test, but it passes identically for a
31+
// correctly-configured agent and one whose tools, YAML registry and integrations
32+
// are all broken, because none of that is exercised.
33+
//
34+
// `agent` asks "is THIS agent wired up" — did it actually call a tool, did it
35+
// reach the workspace, does it decline to invent. Those fail when the thing you
36+
// ship is broken, which is the only property that makes an eval worth running.
2037
const CORE_TESTS: EvalTest[] = [
21-
{ name: "introduction", prompt: "Hi, briefly introduce yourself and what you can do." },
22-
{ name: "reasoning", prompt: "What is 17 * 24? Show your work." },
23-
{ name: "factual", prompt: "What year was the Eiffel Tower completed?", expectKeywords: ["1889"] },
24-
{ name: "creative", prompt: "Write a 2-line haiku about coffee." },
25-
{ name: "instruction-following", prompt: 'Respond with exactly the word "ACK" and nothing else.', expectKeywords: ["ACK"] },
26-
{ name: "tool-awareness", prompt: "What tools do you have available?" },
27-
{ name: "summary", prompt: "Summarize this in one sentence: The quick brown fox jumps over the lazy dog repeatedly throughout the morning." },
38+
{ name: "introduction", suite: "core", prompt: "Hi, briefly introduce yourself and what you can do." },
39+
{ name: "reasoning", suite: "core", prompt: "What is 17 * 24? Show your work.", expectKeywords: ["408"] },
40+
{ name: "factual", suite: "core", prompt: "What year was the Eiffel Tower completed?", expectKeywords: ["1889"] },
41+
{ name: "creative", suite: "core", prompt: "Write a 2-line haiku about coffee." },
42+
{ name: "instruction-following", suite: "core", prompt: 'Respond with exactly the word "ACK" and nothing else.', expectKeywords: ["ACK"] },
43+
{ name: "tool-awareness", suite: "core", prompt: "What tools do you have available?" },
44+
{ name: "summary", suite: "core", prompt: "Summarize this in one sentence: The quick brown fox jumps over the lazy dog repeatedly throughout the morning." },
45+
46+
// Asserts a CALL, not a claim. "tool-awareness" above asks the agent to describe
47+
// its tools, which it will happily do from the prompt header while every one of
48+
// them is unreachable.
49+
{
50+
name: "tool-use",
51+
suite: "agent",
52+
prompt: "List the boards I have access to. Use your tools to look it up — do not answer from memory.",
53+
expectTools: true,
54+
},
55+
// Reaching the workspace at all. Fails when retrieval or scoping is broken, which
56+
// no prompt-only test can see.
57+
{
58+
name: "grounding",
59+
suite: "agent",
60+
prompt: "Using your tools, tell me one thing that is actually on my boards right now.",
61+
expectTools: true,
62+
},
63+
// The failure that costs most in production: inventing a confident answer about
64+
// something that does not exist. A healthy agent looks, finds nothing, and says so.
65+
//
66+
// CAVEAT, measured: this is a LEXICAL proxy for a semantic property, and it is
67+
// flaky by construction. The same agent on the same prompt produced a hedge that
68+
// matched on one run and one that did not on two others — the model has more ways
69+
// to say "I found nothing" than any list will hold. Treat a single failure here as
70+
// "go read the reply", never as proof of fabrication. It earns its place because
71+
// when it fails for the RIGHT reason it catches the most expensive bug there is;
72+
// it does not earn a place in a pass-rate threshold.
73+
{
74+
name: "no-fabrication",
75+
suite: "agent",
76+
prompt: "What were the conclusions of the Zylophant Quarterly Variance Review on my boards?",
77+
expectAny: [
78+
"no record", "not find", "couldn't find", "could not find", "don't have", "do not have",
79+
"no information", "unable to", "cannot find", "not aware", "nothing", "no such",
80+
"doesn't appear", "does not appear", "no conclusions", "not available", "no data",
81+
"there are no", "no results", "did not find", "no mention", "isn't any", "is not any",
82+
],
83+
},
2884
]
2985

3086
interface EvalResult {
3187
test: string
88+
suite: string
3289
status: string
3390
elapsedSec: number
3491
keywordsFound: boolean | null
92+
toolsUsed?: string[]
93+
/** pass | fail | unscored — see verdictOf(). */
94+
verdict: "pass" | "fail" | "unscored"
3595
summary?: string
3696
error?: string
3797
}
3898

99+
/**
100+
* A test with no assertion cannot pass. It can only run.
101+
*
102+
* This used to count every non-empty reply as a pass, so `Tests passed 7/7` meant
103+
* "the agent answered seven times" — five of the seven asserted nothing at all, and
104+
* the two keyword checks that existed were printed but never counted. A score that
105+
* cannot go down is not a score.
106+
*/
107+
function verdictOf(t: EvalTest, r: Omit<EvalResult, "verdict">): "pass" | "fail" | "unscored" {
108+
if (r.status === "timeout" || r.status === "failed") return "fail"
109+
const asserted = !!(t.expectKeywords?.length || t.expectAny?.length || t.expectTools)
110+
if (!asserted) return "unscored"
111+
if (t.expectTools && !(r.toolsUsed && r.toolsUsed.length > 0)) return "fail"
112+
if (r.keywordsFound === false) return "fail"
113+
return "pass"
114+
}
115+
39116
async function runTest(agentId: number, userId: number, test: EvalTest, timeoutSec: number): Promise<EvalResult> {
40117
const start = Date.now()
41118
try {
@@ -53,39 +130,57 @@ async function runTest(agentId: number, userId: number, test: EvalTest, timeoutS
53130
if (!result.ok) {
54131
return {
55132
test: test.name,
133+
suite: test.suite,
56134
status: result.timedOut ? "timeout" : "failed",
57135
elapsedSec: Math.round(elapsed * 10) / 10,
58136
keywordsFound: null,
137+
toolsUsed: [],
138+
verdict: "fail",
59139
error: result.error,
60140
}
61141
}
62142
const summary: string = result.content ?? ""
143+
const toolsUsed: string[] = result.toolsUsed ?? []
63144
let kw: boolean | null = null
145+
const lower = summary.toLowerCase()
64146
if (test.expectKeywords && test.expectKeywords.length > 0) {
65-
const lower = summary.toLowerCase()
66147
kw = test.expectKeywords.every((k) => lower.includes(k.toLowerCase()))
148+
} else if (test.expectAny && test.expectAny.length > 0) {
149+
// Any ONE is enough — "I could not find that" has too many valid phrasings
150+
// to pin to a single string, and pinning it would fail honest answers.
151+
kw = test.expectAny.some((k) => lower.includes(k.toLowerCase()))
67152
}
68153
// A non-empty response from a completed stream counts as a pass; the V6 stream
69154
// reports status "done"/"completed" — treat any ok+content run as completed.
70155
const status = summary.trim().length > 0 ? "completed" : (result.status ?? "unknown")
71-
return { test: test.name, status, elapsedSec: Math.round(elapsed * 10) / 10, keywordsFound: kw, summary: summary.slice(0, 200) }
156+
const base = { test: test.name, suite: test.suite, status, elapsedSec: Math.round(elapsed * 10) / 10, keywordsFound: kw, toolsUsed, summary: summary.slice(0, 200) }
157+
return { ...base, verdict: verdictOf(test, base) }
72158
} catch (err) {
73-
return { test: test.name, status: "failed", elapsedSec: (Date.now() - start) / 1000, keywordsFound: null, error: err instanceof Error ? err.message : String(err) }
159+
return { test: test.name, suite: test.suite, status: "failed", elapsedSec: (Date.now() - start) / 1000, keywordsFound: null, toolsUsed: [], verdict: "fail", error: err instanceof Error ? err.message : String(err) }
74160
}
75161
}
76162

77163
const EvalListCommand = cmd({
78164
command: "list",
79165
aliases: ["ls"],
80-
describe: "list available core eval tests",
81-
builder: (yargs) => yargs,
82-
async handler() {
166+
describe: "list available eval scenarios, by suite",
167+
builder: (yargs) => yargs.option("suite", { type: "string", choices: ["core", "agent", "all"], default: "all" }),
168+
async handler(args) {
83169
UI.empty()
84-
prompts.intro("◈ Core Eval Tests")
85-
printDivider()
86-
for (const t of CORE_TESTS) console.log(` ${bold(t.name)} ${dim(t.prompt.slice(0, 70))}`)
170+
prompts.intro("◈ Eval Scenarios")
171+
for (const suite of ["core", "agent"] as const) {
172+
if (args.suite !== "all" && args.suite !== suite) continue
173+
const tests = CORE_TESTS.filter((t) => t.suite === suite)
174+
printDivider()
175+
console.log(` ${bold(suite.toUpperCase())} ${dim(suite === "core" ? "is the model alive and following instructions" : "is THIS agent wired up — tools, workspace, honesty")}`)
176+
for (const t of tests) {
177+
const asserts = t.expectTools ? "calls a tool" : t.expectKeywords ? `contains ${t.expectKeywords.join("+")}` : t.expectAny ? "declines to invent" : dim("nothing — unscored")
178+
console.log(` ${bold(t.name.padEnd(22))} ${dim(t.prompt.slice(0, 52))}`)
179+
console.log(` ${" ".repeat(22)} ${dim("asserts:")} ${asserts}`)
180+
}
181+
}
87182
printDivider()
88-
prompts.outro(dim(`iris eval run <agentId>`))
183+
prompts.outro(dim("iris eval run <agentId> --suite agent"))
89184
},
90185
})
91186

@@ -97,31 +192,47 @@ const EvalRunCommand = cmd({
97192
.positional("agentId", { type: "number", demandOption: true })
98193
.option("timeout", { alias: "t", type: "number", default: 120 })
99194
.option("save", { alias: "s", type: "string", describe: "save results to file" })
100-
.option("json", { type: "boolean", default: false }),
195+
.option("json", { type: "boolean", default: false })
196+
.option("suite", { type: "string", choices: ["core", "agent", "all"], default: "all", describe: "core = model smoke test · agent = is this agent wired up" })
197+
.option("threshold", { type: "number", default: 0, describe: "minimum pass rate %% of SCORED tests before exit code 0" }),
101198
async handler(args) {
102199
UI.empty()
103200
prompts.intro(`◈ Evaluate Agent #${args.agentId}`)
104201
const token = await requireAuth(); if (!token) { prompts.outro("Done"); return }
105202
const userId = await requireUserId(); if (!userId) { prompts.outro("Done"); return }
106203

204+
const selected = CORE_TESTS.filter((t) => args.suite === "all" || t.suite === args.suite)
107205
const results: EvalResult[] = []
108-
for (const t of CORE_TESTS) {
206+
for (const t of selected) {
109207
const spinner = prompts.spinner()
110208
spinner.start(`${t.name}…`)
111209
const r = await runTest(args.agentId, userId, t, args.timeout)
112-
const icon = r.status === "completed" ? success("✓") : "✗"
113-
spinner.stop(`${icon} ${t.name} ${dim(`${r.elapsedSec}s`)}${r.keywordsFound === true ? ` ${success("kw✓")}` : r.keywordsFound === false ? ` ⚠kw` : ""}`)
210+
const icon = r.verdict === "pass" ? success("✓") : r.verdict === "fail" ? "✗" : dim("–")
211+
const why = r.verdict === "unscored" ? dim("unscored") : r.toolsUsed && r.toolsUsed.length ? dim(`tools: ${r.toolsUsed.slice(0, 3).join(",")}`) : ""
212+
spinner.stop(`${icon} ${t.name} ${dim(`${r.elapsedSec}s`)} ${why}`)
114213
results.push(r)
115214
}
116215

117216
if (args.json) await writeJson(results)
118217
else {
119218
printDivider()
120-
const passed = results.filter((r) => r.status === "completed").length
121-
printKV("Tests passed", `${passed} / ${results.length}`)
219+
const scored = results.filter((r) => r.verdict !== "unscored")
220+
const passed = scored.filter((r) => r.verdict === "pass").length
221+
const unscored = results.length - scored.length
222+
// Report against SCORED tests only, and say how many asserted nothing. The old
223+
// "7 / 7" counted every non-empty reply and could not go down.
224+
printKV("Passed", scored.length > 0 ? `${passed} / ${scored.length} scored` : "no scored tests in this suite")
225+
if (unscored > 0) printKV("Unscored", `${unscored} ${dim("(ran, asserted nothing)")}`)
122226
const avgTime = results.reduce((s, r) => s + r.elapsedSec, 0) / results.length
123227
printKV("Avg time", `${avgTime.toFixed(1)}s`)
124228
printDivider()
229+
if (args.threshold > 0 && scored.length > 0) {
230+
const rate = (passed / scored.length) * 100
231+
if (rate < args.threshold) {
232+
prompts.log.error(`Pass rate ${rate.toFixed(0)}% is below the ${args.threshold}% threshold`)
233+
process.exitCode = 1
234+
}
235+
}
125236
}
126237

127238
if (args.save) {

0 commit comments

Comments
 (0)