Skip to content

Commit 843746d

Browse files
committed
fix(core): bound session title generation with a small output limit
The background title request went out with no max_tokens, so on a local thinking model it reasoned in an unbounded loop for minutes while competing with the main request for CPU (fixes anomalyco#54401). Titles now get a 512-token ceiling through the same outputLimit path as primary and compaction requests; hooks can still change or remove the limit.
1 parent c93aaae commit 843746d

2 files changed

Lines changed: 26 additions & 3 deletions

File tree

‎packages/core/src/session/model-request.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,9 @@ const GENERATION_KEYS = new Set(Object.keys(GenerationOptions.fields))
4848
const OUTPUT_TOKEN_FALLBACK = 32_000
4949
// No reply needs more, however much the model allows.
5050
const OUTPUT_TOKEN_MAX = 256_000
51+
// A title never needs more; without a bound a local thinking model can reason in circles
52+
// for minutes on the background title request while the user waits (#54401).
53+
const TITLE_OUTPUT_MAX = 512
5154
// Prompt text is estimated at about 4 characters per token, which can run low on dense text such as code.
5255
const ESTIMATE_ERROR = 0.15
5356
// Never ask for less; only reachable with automatic compaction off, since it keeps the window from filling this far.
@@ -241,8 +244,9 @@ export const layer = Layer.effect(
241244
const given = new Map(
242245
tools.definitions.map((t) => [{ description: t.description, input: { ...t.inputSchema } }, t] as const),
243246
)
244-
// Hooks see the default output limit and may change or remove it. Titles and generate keep the provider default,
245-
// because their reasoning is hard to budget.
247+
// Hooks see the default output limit and may change or remove it. Titles get a small
248+
// bound so background thinking-model runs cannot outpace the main request; generate
249+
// keeps the provider default, because its reasoning is hard to budget.
246250
const shaped = yield* shape(
247251
{
248252
sessionID: session.id,
@@ -252,7 +256,9 @@ export const layer = Layer.effect(
252256
options:
253257
kind === "primary" || kind === "compaction"
254258
? { maxTokens: outputLimit(model.limit, input.inputTokens) }
255-
: {},
259+
: kind === "title"
260+
? { maxTokens: Math.min(outputLimit(model.limit, input.inputTokens), TITLE_OUTPUT_MAX) }
261+
: {},
256262
},
257263
Object.fromEntries(Array.from(given, ([d, t]) => [t.name, d])),
258264
)

‎packages/core/test/session-title.test.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -674,3 +674,20 @@ it.effect("preserves a manual rename completed while generation is in flight", (
674674
expect((yield* store.get(sessionID))?.title).toBe("Manual title")
675675
}),
676676
)
677+
678+
it.effect("bounds the title request with a small output limit (#54401)", () =>
679+
Effect.gen(function* () {
680+
yield* enableTitleAgent
681+
const sessionID = Session.ID.make("ses_title_limit")
682+
yield* insertSession(sessionID)
683+
yield* prompt(sessionID, "Hello")
684+
685+
const title = yield* SessionTitle.Service
686+
yield* title.generate(sessionID)
687+
688+
expect(requests).toHaveLength(1)
689+
const maxTokens = requests[0]?.generation?.maxTokens
690+
expect(maxTokens).toBeDefined()
691+
expect(maxTokens!).toBeLessThanOrEqual(512)
692+
}),
693+
)

0 commit comments

Comments
 (0)