Skip to content

Commit af81963

Browse files
committed
fix(server): title requests about referenced threads accurately
1 parent 7462f23 commit af81963

4 files changed

Lines changed: 33 additions & 0 deletions

File tree

‎evals/thread-titles/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ pnpm thread-titles --model <codex model> --out runs/titles-new --baseline runs/t
1111
```
1212

1313
- It uses your local Codex login, like the server does.
14+
- `--case <id>` limits the run to one case. Repeat the flag for multiple cases.
1415
- `--initial` titles from the first message only, as a new thread does.
1516
- A run writes `results.json` (titles and latency), `review.json` (titles A and B in random order, with a rubric), and `answer-key.json`. Score `review.json` before you open `answer-key.json`.
1617

‎evals/thread-titles/cases.ts‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,34 @@
11
import type { ThreadTitleMessage } from "@t3tools/provider-core/server/threadTitleContext";
22

33
export const threadTitleEvaluationCases = [
4+
{
5+
id: "referenced-thread-startup",
6+
source: "Maintainer report of chart thread references overriding a startup question",
7+
request: "Explain why one example thread failed to start and another needed a retry.",
8+
previousTitle: "Fix Chart Formatting",
9+
messages: [
10+
{
11+
role: "user",
12+
text: "[xlsx: chart data labels use the cells' number format](t3-context://v1/thread/thread_labels) why could this thread not start up? [Run chart format tests on external target](t3-context://v1/thread/thread_tests) why did i have to say it like this over here to get it working?",
13+
},
14+
],
15+
rubric:
16+
"Name thread startup or prompt/retry behavior. Chart formatting and tests are titles of example threads, not the requested work. Replace the inaccurate previous title.",
17+
},
18+
{
19+
id: "referenced-thread-subject",
20+
source: "Control for maintainer report of referenced thread titles overriding the request",
21+
request: "Continue the actual chart formatting work in a referenced thread.",
22+
previousTitle: "Chart Label Number Formats",
23+
messages: [
24+
{
25+
role: "user",
26+
text: "Continue [xlsx: chart data labels use the cells' number format](t3-context://v1/thread/thread_labels). Make the chart labels inherit the cells' number formats.",
27+
},
28+
],
29+
rubric:
30+
"Name chart label number formatting. Referenced titles can supply the subject when the user explicitly continues that work.",
31+
},
432
{
533
id: "linked-reset-credits",
634
source: "https://github.com/pingdotgg/t3code/pull/10462",

‎evals/thread-titles/run.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ import * as ServerConfig from "../../apps/server/src/config.ts";
3232
const { values } = NodeUtil.parseArgs({
3333
options: {
3434
model: { type: "string" },
35+
case: { type: "string", multiple: true },
3536
out: { type: "string" },
3637
baseline: { type: "string" },
3738
initial: { type: "boolean", default: false },
@@ -70,6 +71,7 @@ await Effect.runPromise(
7071
const review = [];
7172
const answerKey = [];
7273
for (const fixture of threadTitleEvaluationCases) {
74+
if (values.case?.length && !values.case.includes(fixture.id)) continue;
7375
const previous = baseline.find((entry) => entry.id === fixture.id);
7476
if (values.baseline && !previous) throw new Error(`Baseline is missing ${fixture.id}.`);
7577
const firstMessage: ThreadTitleMessage | undefined = fixture.messages.find(

‎packages/provider-core/src/server/textGenerationPrompts.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -391,6 +391,7 @@ Editorial rules:
391391
- 3-8 words, fewer than 40 characters.
392392
- Use a compact noun phrase or clear action phrase.
393393
- Capture the umbrella goal when the request lists several symptoms or steps.
394+
- When asked about thread behavior, never include the referenced titles' topics. "[Fix login] why didn't this thread start?" means "Thread Startup Failure". Use linked topics only for requests about that work.
394395
- Name the product change, not the mock, plan, report, branch, or PR used to produce it.
395396
- Models, subagents, tools, output formats, and monitoring instructions do not belong in the title unless they are themselves the topic.
396397
- For reviews, name what is being reviewed and the relevant concern. Avoid generic titles such as "Review PR 123" when linked or attached context reveals the subject.
@@ -422,6 +423,7 @@ Editorial rules:
422423
- 3-8 words, fewer than 40 characters.
423424
- Use a compact noun phrase or clear action phrase.
424425
- Preserve the umbrella subject when later messages focus on one finding, provider, platform, or implementation detail.
426+
- When asked about thread behavior, never include the referenced titles' topics. "[Fix login] why didn't this thread start?" means "Thread Startup Failure". Use linked topics only for requests about that work.
425427
- A thread progressing through research, planning, implementation, review, CI, merge, and monitoring has usually not changed subjects.
426428
- Ignore deliverables and operations such as mocks, plans, HTML, branches, PRs, tests, CI, commits, merging, and monitoring unless they are the actual topic.
427429
- Models, subagents, tools, output formats, and monitoring instructions do not belong in the title unless they are themselves the topic.

0 commit comments

Comments
 (0)