Skip to content

Commit b06bd16

Browse files
sahrizviclaude
andauthored
fix(session): retry a failed session title on later turns and report why it failed (#1688)
A session whose first title request failed kept its New session placeholder for life and logged only the generic No output generated. The title is now retried on each of the first five user messages with the first message as context, one request in flight per session, and a generated title only ever replaces the placeholder. Each failure is logged with the provider error (name, message, status) and reported as a title_generation_failed telemetry event with no message text; failures before the request, such as a small_model the provider does not list, are reported too. TestLLMServer.title() queues title answers for tests. Docs: telemetry catalogue row and the small_model paragraph on the models page. Closes #1689 Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent f287b68 commit b06bd16

7 files changed

Lines changed: 526 additions & 54 deletions

File tree

‎docs/docs/configure/models.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ altimate supports models from all configured providers. Use the `model` and `sma
1818

1919
The model format is `provider/model-name`.
2020

21+
`small_model` handles the small background requests that are not part of the conversation, such as the session title and prompt enhancement. When it is not set, Altimate Code picks a cheap model from the same provider (Claude Haiku, Gemini Flash or GPT-5 nano when the provider lists one) and otherwise uses the chat model. A `small_model` the provider cannot serve, or does not list, fails only those requests: chat keeps working, while new sessions keep their `New session - <time>` placeholder title until a retry succeeds. Altimate Code retries the title on each of the first five messages of a session and reports every failure in telemetry.
22+
2123
## Browsing Models
2224

2325
In the TUI:

‎docs/docs/reference/telemetry.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ We collect the following categories of events:
2424
| `auth_logout` | A user logs out (provider only) |
2525
| `mcp_server_status` | An MCP server connects, disconnects, or errors (server name and transport) |
2626
| `provider_error` | An AI provider returns an error (error type and HTTP status, but no request content) |
27+
| `title_generation_failed` | A session-title request failed, so the session keeps its `New session - <time>` placeholder until a retry succeeds; retries stop after the fifth user message. Provider, model, `attempt` (1 to 5), error class and type, HTTP status when there was one. No message text. |
2728
| `engine_started` | The native tool engine initializes (version and duration) |
2829
| `engine_error` | The native tool engine fails to start (phase and truncated error) |
2930
| `upgrade_attempted` | A CLI upgrade is attempted (version and method) |
@@ -251,6 +252,7 @@ Event type names use **snake_case** with a `domain_action` pattern:
251252
- `mcp_server_status`, `mcp_server_census` for MCP server lifecycle
252253
- `engine_started`, `engine_error` for native engine events
253254
- `provider_error` for AI provider errors
255+
- `title_generation_failed` for a session-title request the small model did not answer
254256
- `session_forked` for session lifecycle
255257
- `environment_census` for environment snapshot events
256258
- `context_utilization`, `context_overflow_recovered` for context management events

‎packages/opencode/src/altimate/telemetry/index.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -428,6 +428,23 @@ export namespace Telemetry {
428428
error_message: string
429429
http_status?: number
430430
}
431+
// altimate_change start — a session-title request failed, so the session keeps its
432+
// `New session - <time>` placeholder until a later turn's retry succeeds
433+
| {
434+
type: "title_generation_failed"
435+
timestamp: number
436+
session_id: string
437+
provider_id: string
438+
model_id: string
439+
/** Which user message of the session triggered the request: 1 is the first, retries count up. */
440+
attempt: number
441+
/** The error's class name (`AI_APICallError`, `AI_RetryError`, `ProviderModelNotFoundError`, ...), no message text. */
442+
error_type: string
443+
/** Keyword classification from classifyError(). */
444+
error_class: string
445+
http_status?: number
446+
}
447+
// altimate_change end
431448
// DEPRECATED: Python engine eliminated. These event types are retained
432449
// for backward compatibility with existing telemetry dashboards but
433450
// are never fired by the native TypeScript implementation.

‎packages/opencode/src/session/prompt.ts‎

Lines changed: 142 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -4863,6 +4863,42 @@ NOTE: At any point in time through this workflow you should feel free to ask the
48634863
return result
48644864
}
48654865

4866+
// altimate_change start — how many user messages of a session may each trigger a title request
4867+
// while the session still carries its placeholder title.
4868+
const TITLE_MAX_ATTEMPTS = 5
4869+
/** Sessions with a title request in flight: one at a time, so a slow request and the next
4870+
* turn's retry cannot both set the title. */
4871+
const titleInFlight = new Set<string>()
4872+
/** Name, message and HTTP status of a title failure. Covers errors that are not Error instances
4873+
* (openai-compatible in-band errors are strings, Anthropic's are plain objects) and the SDK's
4874+
* RetryError, which carries the status on its last attempt rather than on itself. */
4875+
function describeTitleError(err: unknown): { name: string; message: string; status?: number } {
4876+
const statusOf = (value: unknown) => {
4877+
const o = value as { statusCode?: unknown; lastError?: { statusCode?: unknown } } | undefined
4878+
const code = o?.statusCode ?? o?.lastError?.statusCode
4879+
return typeof code === "number" ? code : undefined
4880+
}
4881+
// The name becomes a telemetry dimension: an identifier-shaped token or nothing.
4882+
const safeName = (value: unknown, fallback: string) =>
4883+
typeof value === "string" && /^[A-Za-z0-9_.:-]{1,64}$/.test(value) ? value : fallback
4884+
if (err instanceof Error) return { name: safeName(err.name, "Error"), message: err.message, status: statusOf(err) }
4885+
if (err && typeof err === "object") {
4886+
const o = err as Record<string, unknown>
4887+
const name = safeName(o.name, safeName(o.type, "object"))
4888+
let message = typeof o.message === "string" ? o.message : ""
4889+
if (!message) {
4890+
try {
4891+
message = JSON.stringify(err).slice(0, 500)
4892+
} catch {
4893+
message = "[object]"
4894+
}
4895+
}
4896+
return { name, message, status: statusOf(err) }
4897+
}
4898+
return { name: typeof err, message: String(err) }
4899+
}
4900+
// altimate_change end
4901+
48664902
async function ensureTitle(input: {
48674903
session: Session.Info
48684904
history: MessageV2.WithParts[]
@@ -4878,10 +4914,16 @@ NOTE: At any point in time through this workflow you should feel free to ask the
48784914
)
48794915
if (firstRealUserIdx === -1) return
48804916

4881-
const isFirst =
4882-
input.history.filter((m) => m.info.role === "user" && !m.parts.every((p) => "synthetic" in p && p.synthetic))
4883-
.length === 1
4884-
if (!isFirst) return
4917+
// altimate_change start — retry while the placeholder stands. Upstream requested a title only
4918+
// on the first user message, so one failed small-model call left `New session - <time>` for the
4919+
// rest of the session (field report, altimate-code 0.12.8). Later turns retry with the same
4920+
// first-message context, bounded so a permanently broken small model does not cost a failed
4921+
// request on every turn.
4922+
const attempt = input.history.filter(
4923+
(m) => m.info.role === "user" && !m.parts.every((p) => "synthetic" in p && p.synthetic),
4924+
).length
4925+
if (attempt > TITLE_MAX_ATTEMPTS) return
4926+
// altimate_change end
48854927

48864928
// Gather all messages up to and including the first real user message for context
48874929
// This includes any shell/subtask executions that preceded the user's first prompt
@@ -4895,56 +4937,103 @@ NOTE: At any point in time through this workflow you should feel free to ask the
48954937

48964938
const agent = await Agent.get("title")
48974939
if (!agent) return
4898-
const model = await iife(async () => {
4899-
// altimate_change start — agent.model carries core ProviderV2.ID/ModelV2.ID brands; re-brand to fork ProviderID/ModelID (identity at runtime)
4900-
if (agent.model)
4901-
return await Provider.getModel(ProviderID.make(agent.model.providerID), ModelID.make(agent.model.modelID))
4902-
// altimate_change end
4903-
return (
4904-
(await Provider.getSmallModel(input.providerID)) ?? (await Provider.getModel(input.providerID, input.modelID))
4905-
)
4906-
})
4907-
const result = await LLM.stream({
4908-
agent,
4909-
user: firstRealUser.info as MessageV2.User,
4910-
system: [],
4911-
small: true,
4912-
tools: {},
4913-
// altimate_change start — title generation is toolless, but without an explicit "none" the
4914-
// historical-tool-stub injection in LLM.stream repopulates `tools` from any tool parts in
4915-
// the context, which both re-declares tools this request cannot use and suppresses the
4916-
// toolless message flattening. compaction.ts passes the same flag for the same reason.
4917-
toolChoice: "none" as const,
4918-
// altimate_change end
4919-
model,
4920-
abort: new AbortController().signal,
4921-
sessionID: input.session.id,
4922-
retries: 2,
4923-
messages: [
4924-
{
4925-
role: "user",
4926-
content: "Generate a title for this conversation:\n",
4927-
},
4928-
...(hasOnlySubtaskParts
4929-
? [{ role: "user" as const, content: subtaskParts.map((p) => p.prompt).join("\n") }]
4930-
: await MessageV2.toModelMessages(contextMessages, model)),
4931-
],
4932-
})
4933-
const text = await Promise.resolve(result.text).catch((err: unknown) => {
4934-
log.error("failed to generate title", { error: err })
4935-
return undefined
4936-
})
4937-
if (text) {
4938-
const cleaned = text
4939-
.replace(/<think>[\s\S]*?<\/think>\s*/g, "")
4940-
.split("\n")
4941-
.map((line: string) => line.trim())
4942-
.find((line: string) => line.length > 0)
4943-
if (!cleaned) return
4944-
4945-
const title = cleaned.length > 100 ? cleaned.substring(0, 97) + "..." : cleaned
4946-
return Session.setTitle({ sessionID: input.session.id, title })
4940+
// altimate_change start — read the stream itself. `result.text` rejects with the SDK's generic
4941+
// "No output generated. Check the stream for errors." and the provider's own error reaches only
4942+
// the stream's error part, so the log never said why a title failed. The model lookup sits in
4943+
// the same try: a `small_model` the provider does not list used to throw out of this
4944+
// fire-and-forget call with no log line and no event. Record the real error and report it, so
4945+
// the next field report is answerable from App Insights.
4946+
if (titleInFlight.has(input.session.id)) return
4947+
titleInFlight.add(input.session.id)
4948+
let text = ""
4949+
let failure: unknown
4950+
let model: Provider.Model | undefined
4951+
try {
4952+
model = await iife(async () => {
4953+
// agent.model carries core ProviderV2.ID/ModelV2.ID brands; re-brand to fork ProviderID/ModelID (identity at runtime)
4954+
if (agent.model)
4955+
return await Provider.getModel(ProviderID.make(agent.model.providerID), ModelID.make(agent.model.modelID))
4956+
return (
4957+
(await Provider.getSmallModel(input.providerID)) ??
4958+
(await Provider.getModel(input.providerID, input.modelID))
4959+
)
4960+
})
4961+
const result = await LLM.stream({
4962+
agent,
4963+
user: firstRealUser.info as MessageV2.User,
4964+
system: [],
4965+
small: true,
4966+
tools: {},
4967+
// title generation is toolless, but without an explicit "none" the historical-tool-stub
4968+
// injection in LLM.stream repopulates `tools` from any tool parts in the context, which both
4969+
// re-declares tools this request cannot use and suppresses the toolless message flattening.
4970+
// compaction.ts passes the same flag for the same reason.
4971+
toolChoice: "none" as const,
4972+
model,
4973+
abort: new AbortController().signal,
4974+
sessionID: input.session.id,
4975+
retries: 2,
4976+
messages: [
4977+
{
4978+
role: "user",
4979+
content: "Generate a title for this conversation:\n",
4980+
},
4981+
...(hasOnlySubtaskParts
4982+
? [{ role: "user" as const, content: subtaskParts.map((p) => p.prompt).join("\n") }]
4983+
: await MessageV2.toModelMessages(contextMessages, model)),
4984+
],
4985+
})
4986+
for await (const part of result.fullStream) {
4987+
if (part.type === "text-delta") text += part.text
4988+
else if (part.type === "error") failure = part.error
4989+
}
4990+
} catch (err) {
4991+
failure = err
4992+
} finally {
4993+
titleInFlight.delete(input.session.id)
4994+
}
4995+
const cleaned = text
4996+
.replace(/<think>[\s\S]*?<\/think>\s*/g, "")
4997+
.split("\n")
4998+
.map((line: string) => line.trim())
4999+
.find((line: string) => line.length > 0)
5000+
// An answer with no usable line (empty, or thinking only) is a failed attempt too, so it is
5001+
// reported rather than silently using up one of the five.
5002+
if (failure !== undefined || !cleaned) {
5003+
const err = failure ?? new Error(text.trim() ? "the model returned no title line" : "the model returned no text")
5004+
const described = describeTitleError(err)
5005+
const providerID = model?.providerID ?? input.providerID
5006+
const modelID = model?.id ?? input.modelID
5007+
// Name, message and status only: a provider error object can carry the request body.
5008+
log.error("failed to generate title", {
5009+
errorType: described.name,
5010+
message: described.message.slice(0, 500),
5011+
status: described.status,
5012+
attempt,
5013+
providerID,
5014+
modelID,
5015+
})
5016+
Telemetry.track({
5017+
type: "title_generation_failed",
5018+
timestamp: Date.now(),
5019+
session_id: input.session.id,
5020+
provider_id: providerID,
5021+
model_id: modelID,
5022+
attempt,
5023+
error_type: described.name,
5024+
error_class: Telemetry.classifyError(described.message),
5025+
...(described.status !== undefined ? { http_status: described.status } : {}),
5026+
})
5027+
return
49475028
}
5029+
5030+
const title = cleaned.length > 100 ? cleaned.substring(0, 97) + "..." : cleaned
5031+
// The user may have named the session while the request was in flight: a generated title only
5032+
// ever replaces the placeholder, and when that cannot be checked the title is left for a retry.
5033+
const current = await Session.get(input.session.id).catch(() => undefined)
5034+
if (!current || !Session.isDefaultTitle(current.title)) return
5035+
// altimate_change end
5036+
return Session.setTitle({ sessionID: input.session.id, title })
49485037
}
49495038

49505039
// altimate_change start — Effect Context.Service facade over the existing Promise namespace.

‎packages/opencode/test/lib/llm-server.ts‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -616,6 +616,8 @@ namespace TestLLMServer {
616616
readonly url: string
617617
readonly push: (...input: (Item | Reply)[]) => Effect.Effect<void>
618618
readonly pushMatch: (match: Match, ...input: (Item | Reply)[]) => Effect.Effect<void>
619+
/** Answers for the next title requests, in order; the automatic "E2E Title" resumes after them. */
620+
readonly title: (...input: (Item | Reply)[]) => Effect.Effect<void>
619621
readonly textMatch: (match: Match, value: string, opts?: { usage?: Usage }) => Effect.Effect<void>
620622
readonly toolMatch: (match: Match, name: string, input: unknown) => Effect.Effect<void>
621623
readonly text: (value: string, opts?: { usage?: Usage }) => Effect.Effect<void>
@@ -648,6 +650,7 @@ export class TestLLMServer extends Context.Service<TestLLMServer, TestLLMServer.
648650

649651
let hits: Hit[] = []
650652
let list: Queue[] = []
653+
let titles: Item[] = [] // altimate_change — answers for title requests, see handle()
651654
let waits: Wait[] = []
652655
let misses: Hit[] = []
653656

@@ -679,8 +682,22 @@ export class TestLLMServer extends Context.Service<TestLLMServer, TestLLMServer.
679682
const body = yield* req.json.pipe(Effect.orElseSucceed(() => ({})))
680683
const current = hit(req.originalUrl, body, { ...req.headers })
681684
if (isTitleRequest(body)) {
685+
// altimate_change start — a queued title item answers the request instead of the automatic
686+
// "E2E Title", so a test can make title generation fail. Kept apart from the main queue:
687+
// a title request carries the first user message, so body matchers would catch it too.
688+
const queued = titles.shift()
682689
hits = [...hits, current]
683690
yield* notify()
691+
if (queued) {
692+
if (queued.type !== "sse") return fail(queued)
693+
if (mode === "responses") return send(responses(queued, modelFrom(body)))
694+
if (queued.reset) {
695+
yield* reset(queued)
696+
return HttpServerResponse.empty()
697+
}
698+
return send(queued)
699+
}
700+
// altimate_change end
684701
const auto: Sse = { type: "sse", head: [role()], tail: [textLine("E2E Title"), finishLine("stop")] }
685702
if (mode === "responses") return send(responses(auto, modelFrom(body)))
686703
return send(auto)
@@ -720,6 +737,12 @@ export class TestLLMServer extends Context.Service<TestLLMServer, TestLLMServer.
720737
pushMatch: Effect.fn("TestLLMServer.pushMatch")(function* (match: Match, ...input: (Item | Reply)[]) {
721738
queueMatch(match, ...input)
722739
}),
740+
// altimate_change start — answers for the next title requests, in order; the automatic
741+
// "E2E Title" resumes once they are used up
742+
title: Effect.fn("TestLLMServer.title")(function* (...input: (Item | Reply)[]) {
743+
titles = [...titles, ...input.map((value) => item(value))]
744+
}),
745+
// altimate_change end
723746
textMatch: Effect.fn("TestLLMServer.textMatch")(function* (
724747
match: Match,
725748
value: string,
@@ -767,6 +790,7 @@ export class TestLLMServer extends Context.Service<TestLLMServer, TestLLMServer.
767790
reset: Effect.sync(() => {
768791
hits = []
769792
list = []
793+
titles = [] // altimate_change
770794
waits = []
771795
misses = []
772796
}),

0 commit comments

Comments
 (0)