Skip to content

Commit cf9c05f

Browse files
t3dotggclaude
andcommitted
test(evals): test-signal check is cut to 88 words, measured on cheap models
Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 7618c77 commit cf9c05f

10 files changed

Lines changed: 632 additions & 43 deletions

File tree

‎.macroscope/check-run-agents/test-signal.md‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,16 +22,6 @@ conclusion: neutral
2222

2323
Review only the tests this pull request adds or grows. Tests it moves, renames, or deletes are not findings. The rules are in `docs/internals/testing.md` under `## What to test`; read that section first and hold new tests to it.
2424

25-
## Checklist
26-
27-
Flag a new or grown test when one of these is true:
28-
29-
- It tests what types already guarantee, trivial logic, copy text, a smoke run, mock calls that mirror the implementation, callback wiring, or props rendered to static markup.
30-
- An existing test or table in the same file already covers the behavior. The case belongs there as a row or an assertion.
31-
- The same case is also tested at another layer, for example in a service test and its wrapper test, at each call site of one helper, or in web and mobile copies of the same logic. Keep one copy at the lowest layer that fails on `main`.
32-
33-
## Reporting
34-
35-
Post each finding as a brief inline comment on the new test. Name the rule and give one fix: fold the case into the named test or table, delete it, or keep the copy at the named layer and delete the others. Ask only for fewer or smaller tests, never for more coverage.
25+
Post each finding as a brief inline comment on the new test with one fix. Ask only for fewer or smaller tests, never for more coverage.
3626

3727
When there are no findings, make the entire final response exactly `All clear` on one line with nothing else.

‎evals/README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,11 @@ An eval runs a real model against T3 Code's real text or code and scores what th
44

55
Write or run an eval when a change affects what a model sees or does: agent instructions, MCP tool descriptions and schemas, tool results, generated titles, or the guidance an agent gets for a feature such as Device Manager.
66

7-
| Eval | Measures |
8-
| ------------------------------- | ------------------------------------------------------------ |
9-
| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly in fake threads. |
10-
| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
7+
| Eval | Measures |
8+
| ------------------------------- | --------------------------------------------------------------------------- |
9+
| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly in fake threads. |
10+
| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
11+
| [test-signal](test-signal/) | Whether a cheap model running the test-signal check flags low-signal tests. |
1112

1213
## Run an eval
1314

‎evals/mcp-tool-use/run.ts‎

Lines changed: 1 addition & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ import {
5050
CLAUDE_PROVIDER,
5151
makeClaudeQueryOptions,
5252
} from "../../apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts";
53+
import { proxyConfig } from "../shared/proxy.ts";
5354
import { changeRequestsCreatedBy } from "../../apps/server/src/orchestration-v2/PullRequestSyncReactor.ts";
5455
/** Agent models by id, with the display name a harness gives them. */
5556
const MODELS = {
@@ -1342,34 +1343,6 @@ const callCodex = Effect.fn("evaluateMcpToolUse.callCodex")(function* (
13421343
};
13431344
});
13441345

1345-
// Read credentials into memory only. Python's TOML parser avoids another runner dependency.
1346-
const decodeProxyConfig = Schema.decodeUnknownSync(
1347-
Schema.Struct({ url: Schema.String, headers: Schema.Record(Schema.String, Schema.String) }),
1348-
);
1349-
const proxyConfig = () =>
1350-
decodeProxyConfig(
1351-
JSON.parse(
1352-
NodeChildProcess.execFileSync(
1353-
"python3",
1354-
[
1355-
"-c",
1356-
`
1357-
import json, os, pathlib, tomllib
1358-
home = pathlib.Path.home() / '.codex'
1359-
p = tomllib.loads((home / 'config.toml').read_text())['model_providers']['vibeproxy']
1360-
h = p.get('http_headers', {}).copy()
1361-
h.update({k: os.environ[v] for k,v in p.get('env_http_headers', {}).items()})
1362-
key = os.environ[p['env_key']] if p.get('env_key') else p.get('experimental_bearer_token')
1363-
if not key and p.get('requires_openai_auth'): key = json.loads((home / 'auth.json').read_text()).get('OPENAI_API_KEY')
1364-
if key: h['Authorization'] = 'Bearer ' + key
1365-
print(json.dumps({'url': p['base_url'].rstrip('/'), 'headers': h}))
1366-
`,
1367-
],
1368-
{ encoding: "utf8" },
1369-
),
1370-
),
1371-
);
1372-
13731346
const postModel = Effect.fn("evaluateMcpToolUse.postModel")(function* (
13741347
apiKey: string,
13751348
body: unknown,

‎evals/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
"mcp-tool-use": "node --env-file-if-exists=../.env mcp-tool-use/run.ts",
77
"mcp-tool-use:trim-capture": "node mcp-tool-use/trim-capture.ts",
88
"thread-titles": "node --env-file-if-exists=../.env thread-titles/run.ts",
9+
"test-signal": "node --env-file-if-exists=../.env test-signal/run.ts",
910
"typecheck": "tsc --noEmit",
1011
"test": "vp test run --config ../vite.config.ts --dir ."
1112
},

‎evals/shared/proxy.ts‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
// @effect-diagnostics nodeBuiltinImport:off
2+
import * as NodeChildProcess from "node:child_process";
3+
import * as Schema from "effect/Schema";
4+
5+
// Read credentials into memory only. Python's TOML parser avoids another runner dependency.
6+
const decodeProxyConfig = Schema.decodeUnknownSync(
7+
Schema.Struct({ url: Schema.String, headers: Schema.Record(Schema.String, Schema.String) }),
8+
);
9+
/** Reads the `vibeproxy` CLIProxyAPI provider from `~/.codex/config.toml`. */
10+
export const proxyConfig = () =>
11+
decodeProxyConfig(
12+
JSON.parse(
13+
NodeChildProcess.execFileSync(
14+
"python3",
15+
[
16+
"-c",
17+
`
18+
import json, os, pathlib, tomllib
19+
home = pathlib.Path.home() / '.codex'
20+
p = tomllib.loads((home / 'config.toml').read_text())['model_providers']['vibeproxy']
21+
h = p.get('http_headers', {}).copy()
22+
h.update({k: os.environ[v] for k,v in p.get('env_http_headers', {}).items()})
23+
key = os.environ[p['env_key']] if p.get('env_key') else p.get('experimental_bearer_token')
24+
if not key and p.get('requires_openai_auth'): key = json.loads((home / 'auth.json').read_text()).get('OPENAI_API_KEY')
25+
if key: h['Authorization'] = 'Bearer ' + key
26+
print(json.dumps({'url': p['base_url'].rstrip('/'), 'headers': h}))
27+
`,
28+
],
29+
{ encoding: "utf8" },
30+
),
31+
),
32+
);

‎evals/test-signal/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Test signal
2+
3+
Runs the Macroscope `test-signal` check ([`.macroscope/check-run-agents/test-signal.md`](../../.macroscope/check-run-agents/test-signal.md)) on real and synthetic test diffs, and checks that a cheap model flags low-signal tests and leaves good ones alone. Use it to cut the check to the fewest words that still work.
4+
5+
The model gets the check body, the PR's test diff, `read_file` and `grep` at the PR head, and `post_comment`. The wrapper prompt around the body is a stand-in for Macroscope's, which is not public.
6+
7+
## Run
8+
9+
```sh
10+
cd evals
11+
pnpm test-signal --out runs/baseline
12+
pnpm test-signal --out runs/cut --prompt /path/to/variant.md --case new-table-row
13+
```
14+
15+
- Models go through the `vibeproxy` CLIProxyAPI provider in `~/.codex/config.toml`. Default models: `gpt-6.1-terra` and `gpt-6-luna`. Pass `--model` once per model to change them.
16+
- Flags: `--prompt <file>` (a variant of the check, default the real one), `--case <id>` (repeatable), `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (default 6), `--reasoning <level>` (default `low`, as the check runs).
17+
- A bad case passes when a comment lands inside a target test. A guard case passes when the check posts nothing and replies exactly `All clear`.
18+
- `summary.md` gives the prompt hash and word count, then per model: bad cases caught, guards clean, errors, and tokens.
19+
- Cost: proxy quota only, about 0.5M tokens for a full run on both models.
20+
21+
## Add a case
22+
23+
Add an entry to `cases.ts`. A `real` case names a merged commit and its test paths. A `synthetic` case inserts one test after an anchor string in a real file. Bad cases list the titles of the tests to flag.

‎evals/test-signal/cases.ts‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
/**
2+
* Pull request test diffs for the test-signal check. A `real` case replays test files from a
3+
* merged PR. A `synthetic` case inserts one test into a real file at `rev`, for a pattern with no
4+
* clean real example.
5+
*/
6+
export type Case = {
7+
readonly id: string;
8+
readonly kind: "bad" | "guard";
9+
readonly source: string;
10+
readonly rev: string;
11+
/** Titles of the tests the check must flag. Guard cases expect no comment and `All clear`. */
12+
readonly targets: ReadonlyArray<{ readonly path: string; readonly title: string }>;
13+
} & (
14+
| { readonly type: "real"; readonly paths: ReadonlyArray<string> }
15+
| {
16+
readonly type: "synthetic";
17+
readonly path: string;
18+
readonly after: string;
19+
readonly insert: string;
20+
}
21+
);
22+
23+
const base = "cb7e3814d71c8fcdc5b803c660c29b7d7e39be96";
24+
25+
export const cases: ReadonlyArray<Case> = [
26+
{
27+
id: "copy-text-at-every-call-site",
28+
kind: "bad",
29+
source: "#23999 checks the PR or MR label in 11 files",
30+
type: "real",
31+
rev: "218e5c09df",
32+
paths: [
33+
"packages/client-runtime/src/state/gitActions.test.ts",
34+
"apps/web/src/lib/composerContextRecords.test.ts",
35+
"apps/mobile/src/lib/composerContext.test.ts",
36+
],
37+
targets: [
38+
{
39+
path: "packages/client-runtime/src/state/gitActions.test.ts",
40+
title: "says PR for a GitHub ref",
41+
},
42+
{
43+
path: "packages/client-runtime/src/state/gitActions.test.ts",
44+
title: "says MR for a GitLab ref",
45+
},
46+
{
47+
path: "packages/client-runtime/src/state/gitActions.test.ts",
48+
title: "names the host's change request in the default-branch dialog",
49+
},
50+
{
51+
path: "apps/web/src/lib/composerContextRecords.test.ts",
52+
title: "names the change request the way its host does",
53+
},
54+
{
55+
path: "apps/mobile/src/lib/composerContext.test.ts",
56+
title: "describes the change request with its host's term",
57+
},
58+
],
59+
},
60+
{
61+
id: "same-case-in-wrapper-test",
62+
kind: "bad",
63+
source: "#22452 added the same two cases to threadAccess and OrchestratorMcpService",
64+
type: "real",
65+
rev: "b260a7cf69",
66+
paths: [
67+
"apps/server/src/mcp/threadAccess.test.ts",
68+
"apps/server/src/mcp/OrchestratorMcpService.test.ts",
69+
],
70+
// Either copy counts: the check must notice the duplicate. Which copy it keeps is secondary.
71+
targets: [
72+
{
73+
path: "apps/server/src/mcp/threadAccess.test.ts",
74+
title: "points a bare uuid at the mcp: thread it belongs to",
75+
},
76+
{
77+
path: "apps/server/src/mcp/threadAccess.test.ts",
78+
title: "keeps the plain message for an id that matches nothing",
79+
},
80+
{
81+
path: "apps/server/src/mcp/OrchestratorMcpService.test.ts",
82+
title: "points a bare uuid at the mcp: thread it belongs to",
83+
},
84+
{
85+
path: "apps/server/src/mcp/OrchestratorMcpService.test.ts",
86+
title: "keeps the plain message for an id that matches nothing",
87+
},
88+
],
89+
},
90+
{
91+
id: "new-it-for-a-table-row",
92+
kind: "bad",
93+
source: "Synthetic: a new `it` next to the table that already covers it",
94+
type: "synthetic",
95+
rev: base,
96+
path: "packages/shared/src/markdownLinks.test.ts",
97+
after: " expect(parseMarkdownFileLink(href)?.path).toBe(path);\n });\n",
98+
insert:
99+
'\n it("recognizes a Windows path with forward slashes as a file", () => {\n expect(parseMarkdownFileLink("C:/repo/src/app.ts")?.path).toBe("C:/repo/src/app.ts");\n });\n',
100+
targets: [
101+
{
102+
path: "packages/shared/src/markdownLinks.test.ts",
103+
title: "recognizes a Windows path with forward slashes as a file",
104+
},
105+
],
106+
},
107+
{
108+
id: "mock-call-mirror",
109+
kind: "bad",
110+
source: "Synthetic: asserts the mock call the code makes, already covered above it",
111+
type: "synthetic",
112+
rev: base,
113+
path: "packages/client-runtime/src/state/threadRequests.test.ts",
114+
after: " expect(send).toHaveBeenCalledOnce();\n });\n",
115+
insert:
116+
'\n it("calls respond with decline", async () => {\n const respond = vi.fn<ApprovalResponder>(async () => AsyncResult.success(undefined));\n await declinePendingApprovals([approval], respond);\n expect(respond).toHaveBeenCalledWith(requestId, "decline");\n expect(respond).toHaveBeenCalledTimes(1);\n });\n',
117+
targets: [
118+
{
119+
path: "packages/client-runtime/src/state/threadRequests.test.ts",
120+
title: "calls respond with decline",
121+
},
122+
],
123+
},
124+
{
125+
id: "type-guaranteed",
126+
kind: "bad",
127+
source: "Synthetic: checks a return type the signature already declares",
128+
type: "synthetic",
129+
rev: base,
130+
path: "packages/shared/src/markdownLinks.test.ts",
131+
after: " expect(isMarkdownFileLinkLabel(label, href)).toBe(expected);\n });\n",
132+
insert:
133+
'\n it("returns a boolean", () => {\n expect(typeof isMarkdownFileLinkLabel("a.ts", "/repo/a.ts")).toBe("boolean");\n });\n',
134+
targets: [{ path: "packages/shared/src/markdownLinks.test.ts", title: "returns a boolean" }],
135+
},
136+
{
137+
id: "deadlock-race-fix",
138+
kind: "guard",
139+
source: "#22094 run metrics read lineage inside the transaction so a writer cannot deadlock",
140+
type: "real",
141+
rev: "f8e4834861",
142+
paths: ["apps/server/src/orchestration-v2/RunMetrics.test.ts"],
143+
targets: [],
144+
},
145+
{
146+
id: "clone-retry-fix",
147+
kind: "guard",
148+
source: "#23170 clones retry over HTTPS when the SSH key is refused",
149+
type: "real",
150+
rev: "a49d18cf3b",
151+
paths: ["apps/server/src/sourceControl/SourceControlRepositoryService.test.ts"],
152+
targets: [],
153+
},
154+
{
155+
id: "deleted-tests",
156+
kind: "guard",
157+
source: "#22592 pruned low-value tests",
158+
type: "real",
159+
rev: "a2f4df0412",
160+
paths: [
161+
"apps/server/src/workspace/WorkspaceEntries.test.ts",
162+
"apps/server/src/workspace/WorkspacePaths.test.ts",
163+
],
164+
targets: [],
165+
},
166+
{
167+
id: "new-table-row",
168+
kind: "guard",
169+
source: "Synthetic: the right way to add the Windows path case",
170+
type: "synthetic",
171+
rev: base,
172+
path: "packages/shared/src/markdownLinks.test.ts",
173+
after: ' ["file://localhost/home/me/notes.md", "/home/me/notes.md"],\n',
174+
insert: ' ["C:/repo/src/app.ts", "C:/repo/src/app.ts"],\n',
175+
targets: [],
176+
},
177+
];

‎evals/test-signal/grade.ts‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
export type Comment = { readonly path: string; readonly line: number; readonly body: string };
2+
3+
const testStart = /^\s*(it|test|describe)(\.\w+)*\(/;
4+
5+
/** 1-based line range of the test whose title is `title`, up to the next test or describe. */
6+
export const testRange = (source: string, title: string) => {
7+
const lines = source.split("\n");
8+
const start = lines.findIndex(
9+
(line) => line.includes(`"${title}"`) || line.includes(`'${title}'`),
10+
);
11+
if (start < 0) return undefined;
12+
const next = lines.findIndex((line, index) => index > start && testStart.test(line));
13+
return { start: start + 1, end: next < 0 ? lines.length : next };
14+
};
15+
16+
// Models often anchor a comment on the blank line or brace just above the test.
17+
const slack = 2;
18+
19+
/**
20+
* A bad case passes when a comment lands inside a target test. A guard case passes when the
21+
* check posts nothing and ends with exactly `All clear`, as Macroscope requires.
22+
*/
23+
export const grade = (input: {
24+
readonly kind: "bad" | "guard";
25+
readonly targets: ReadonlyArray<{
26+
readonly path: string;
27+
readonly start: number;
28+
readonly end: number;
29+
}>;
30+
readonly comments: ReadonlyArray<Comment>;
31+
readonly final: string;
32+
}) => {
33+
if (input.kind === "guard") {
34+
if (input.comments.length > 0)
35+
return { pass: false, reason: `${input.comments.length} comment(s)` };
36+
if (input.final.trim() !== "All clear")
37+
return { pass: false, reason: "final reply is not `All clear`" };
38+
return { pass: true, reason: "all clear" };
39+
}
40+
const hit = input.comments.some((comment) =>
41+
input.targets.some(
42+
(target) =>
43+
comment.path === target.path &&
44+
comment.line >= target.start - slack &&
45+
comment.line <= target.end,
46+
),
47+
);
48+
return hit
49+
? { pass: true, reason: "flagged a target" }
50+
: { pass: false, reason: `missed (${input.comments.length} comment(s) elsewhere)` };
51+
};

‎evals/test-signal/run.test.ts‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
import { describe, expect, it } from "vite-plus/test";
2+
import { grade, testRange } from "./grade.ts";
3+
4+
const source = [
5+
'describe("x", () => {',
6+
' it("first", () => {',
7+
" expect(1).toBe(1);",
8+
" });",
9+
"",
10+
' it("second", () => {});',
11+
"});",
12+
].join("\n");
13+
14+
describe("grade", () => {
15+
const target = { path: "a.test.ts", ...testRange(source, "first")! };
16+
const comment = (line: number) => ({ path: "a.test.ts", line, body: "fold it" });
17+
18+
it.each([
19+
["bad, comment inside the target", "bad", [comment(3)], "", true],
20+
["bad, comment on the next test", "bad", [comment(6)], "", false],
21+
["guard, nothing posted", "guard", [], "All clear", true],
22+
["guard, nothing posted but extra text", "guard", [], "All clear. Nice tests.", false],
23+
["guard, a comment", "guard", [comment(3)], "All clear", false],
24+
] as const)("%s", (_name, kind, comments, final, pass) => {
25+
expect(grade({ kind, targets: [target], comments, final }).pass).toBe(pass);
26+
});
27+
});

0 commit comments

Comments
 (0)