You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cf8c8d1
Browse filesBrowse the repository at this point in the historyBrowse files
fix(eval): a score that cannot go down is not a score
`iris eval run` reported "Tests passed 7/7". It counted every non-empty reply as
a pass:
const passed = results.filter((r) => r.status === "completed").length
// "completed" = summary.trim().length > 0
Four of the seven scenarios asserted NOTHING — introduction, creative,
tool-awareness, summary — so they passed by existing. The two that did have
keyword checks had their results printed and then excluded from the count. The
number could not go down, which made it a liveness probe wearing a score's
clothes.
Two changes.
1. A test with no assertion cannot pass; it can only run. Verdicts are now
pass | fail | unscored, and the summary reads "3 / 3 scored · 4 unscored
(ran, asserted nothing)" instead of 7/7. Same agent, same run, honest number.
2. A second suite that tests THIS agent rather than the model underneath it.
`core` (Eiffel Tower, 17*24, a haiku) passes identically whether the agent's
tools, YAML registry and integrations work or are entirely broken, because
none of it is exercised. `agent` asserts:
tool-use the agent CALLED a tool — toolsUsed non-empty. The old
"what tools do you have?" is a self-report the agent will
happily give from its prompt header while every one of them
is unreachable.
grounding it reached the workspace at all
no-fabrication it declines to invent when asked about something that does
not exist
Verified against PATTY (#420), not asserted: tool-use called
Pathways_get_firm_directory, grounding called SearchKnowledgeBaseTool. And it
fails honestly — agent #11 returned "Resource not found" and scored 0/3 rather
than passing vacuously.
no-fabrication is documented in-code as flaky BY CONSTRUCTION, because it is a
lexical proxy for a semantic property: the same agent on the same prompt hedged
in wording my list matched once and missed twice. The caveat says to read the
reply on a failure rather than conclude fabrication, and not to put it in a
threshold. Better to ship the limitation written down than a check that looks
deterministic and is not.
No MCP change. `iris_run` already executes any CLI command, so the MCP server
stays at seven tools and picks this up for free.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JQ9915AheUUs2qL2D2u62K
@@ -13,29 +13,106 @@ import { writeFileSync } from "fs"
13
13
14
14
interfaceEvalTest{
15
15
name: string
16
+
suite: "core"|"agent"
16
17
prompt: string
18
+
/** All of these must appear in the reply. */
17
19
expectKeywords?: string[]
20
+
/** At least ONE of these must appear — for checks with many valid phrasings. */
21
+
expectAny?: string[]
22
+
/** The agent must have actually CALLED a tool, not merely described having them. */
23
+
expectTools?: boolean
18
24
}
19
25
26
+
// Two suites, because they answer different questions and only one of them is
27
+
// about IRIS.
28
+
//
29
+
// `core` asks "is the model alive and following instructions" — Eiffel Tower,
30
+
// 17*24, a haiku. Useful as a smoke test, but it passes identically for a
31
+
// correctly-configured agent and one whose tools, YAML registry and integrations
32
+
// are all broken, because none of that is exercised.
33
+
//
34
+
// `agent` asks "is THIS agent wired up" — did it actually call a tool, did it
35
+
// reach the workspace, does it decline to invent. Those fail when the thing you
36
+
// ship is broken, which is the only property that makes an eval worth running.
20
37
constCORE_TESTS: EvalTest[]=[
21
-
{name: "introduction",prompt: "Hi, briefly introduce yourself and what you can do."},
22
-
{name: "reasoning",prompt: "What is 17 * 24? Show your work."},
23
-
{name: "factual",prompt: "What year was the Eiffel Tower completed?",expectKeywords: ["1889"]},
24
-
{name: "creative",prompt: "Write a 2-line haiku about coffee."},
25
-
{name: "instruction-following",prompt: 'Respond with exactly the word "ACK" and nothing else.',expectKeywords: ["ACK"]},
26
-
{name: "tool-awareness",prompt: "What tools do you have available?"},
27
-
{name: "summary",prompt: "Summarize this in one sentence: The quick brown fox jumps over the lazy dog repeatedly throughout the morning."},
38
+
{name: "introduction",suite: "core",prompt: "Hi, briefly introduce yourself and what you can do."},
39
+
{name: "reasoning",suite: "core",prompt: "What is 17 * 24? Show your work.",expectKeywords: ["408"]},
40
+
{name: "factual",suite: "core",prompt: "What year was the Eiffel Tower completed?",expectKeywords: ["1889"]},
41
+
{name: "creative",suite: "core",prompt: "Write a 2-line haiku about coffee."},
42
+
{name: "instruction-following",suite: "core",prompt: 'Respond with exactly the word "ACK" and nothing else.',expectKeywords: ["ACK"]},
43
+
{name: "tool-awareness",suite: "core",prompt: "What tools do you have available?"},
44
+
{name: "summary",suite: "core",prompt: "Summarize this in one sentence: The quick brown fox jumps over the lazy dog repeatedly throughout the morning."},
45
+
46
+
// Asserts a CALL, not a claim. "tool-awareness" above asks the agent to describe
47
+
// its tools, which it will happily do from the prompt header while every one of
48
+
// them is unreachable.
49
+
{
50
+
name: "tool-use",
51
+
suite: "agent",
52
+
prompt: "List the boards I have access to. Use your tools to look it up — do not answer from memory.",
53
+
expectTools: true,
54
+
},
55
+
// Reaching the workspace at all. Fails when retrieval or scoping is broken, which
56
+
// no prompt-only test can see.
57
+
{
58
+
name: "grounding",
59
+
suite: "agent",
60
+
prompt: "Using your tools, tell me one thing that is actually on my boards right now.",
61
+
expectTools: true,
62
+
},
63
+
// The failure that costs most in production: inventing a confident answer about
64
+
// something that does not exist. A healthy agent looks, finds nothing, and says so.
65
+
//
66
+
// CAVEAT, measured: this is a LEXICAL proxy for a semantic property, and it is
67
+
// flaky by construction. The same agent on the same prompt produced a hedge that
68
+
// matched on one run and one that did not on two others — the model has more ways
69
+
// to say "I found nothing" than any list will hold. Treat a single failure here as
70
+
// "go read the reply", never as proof of fabrication. It earns its place because
71
+
// when it fails for the RIGHT reason it catches the most expensive bug there is;
72
+
// it does not earn a place in a pass-rate threshold.
73
+
{
74
+
name: "no-fabrication",
75
+
suite: "agent",
76
+
prompt: "What were the conclusions of the Zylophant Quarterly Variance Review on my boards?",
77
+
expectAny: [
78
+
"no record","not find","couldn't find","could not find","don't have","do not have",
79
+
"no information","unable to","cannot find","not aware","nothing","no such",
80
+
"doesn't appear","does not appear","no conclusions","not available","no data",
81
+
"there are no","no results","did not find","no mention","isn't any","is not any",
82
+
],
83
+
},
28
84
]
29
85
30
86
interfaceEvalResult{
31
87
test: string
88
+
suite: string
32
89
status: string
33
90
elapsedSec: number
34
91
keywordsFound: boolean|null
92
+
toolsUsed?: string[]
93
+
/** pass | fail | unscored — see verdictOf(). */
94
+
verdict: "pass"|"fail"|"unscored"
35
95
summary?: string
36
96
error?: string
37
97
}
38
98
99
+
/**
100
+
* A test with no assertion cannot pass. It can only run.
101
+
*
102
+
* This used to count every non-empty reply as a pass, so `Tests passed 7/7` meant
103
+
* "the agent answered seven times" — five of the seven asserted nothing at all, and
104
+
* the two keyword checks that existed were printed but never counted. A score that
console.log(` ${bold(suite.toUpperCase())}${dim(suite==="core" ? "is the model alive and following instructions" : "is THIS agent wired up — tools, workspace, honesty")}`)
176
+
for(consttoftests){
177
+
constasserts=t.expectTools ? "calls a tool" : t.expectKeywords ? `contains ${t.expectKeywords.join("+")}` : t.expectAny ? "declines to invent" : dim("nothing — unscored")
0 commit comments