Skip to content

Commit 1586f59

Browse files
anandgupta42anandgupta42
committed
fix: clamp the output-token reservation against the context window
`ProviderTransform.maxOutputTokens` is a per-model ceiling that never reads `limit.context`, so on a model where the prompt and the completion share one window a large system prompt pushed `input + reservation` past that window and the provider rejected the request with a hard 400 before generating anything. Measured case: a ~52,180-token prompt plus the 16,384-token reservation on a 65,536-token window. Compaction cannot cover this. `Compaction.isOverflow` runs off the previous assistant message's token counts, so there is nothing to check on the first request of a session, and compaction only shortens conversation messages — never the system prompt, which is where the whole overflow lives here. The failure was also not classified as an overflow: the provider's wording ("maximum context length **of** 65536 tokens") matches no pattern in `OVERFLOW_PATTERNS`, the status is 400 rather than 413, and the body carries no `context_length_exceeded` code. It surfaced as a generic non-retryable `APIError` and the session died showing raw provider text. - add `ProviderTransform.clampOutputTokens`, which shrinks the reservation so `input + reservation` fits the window, and `estimateInputTokens` to size the prompt about to be sent - throw `OutputTokenBudgetError` before the request when clamping cannot leave `OUTPUT_TOKEN_FLOOR` (1,024) tokens, naming the prompt size, the requested reservation, the window, and the three ways to fix it - keep a 2% (minimum 512-token) margin, since the input count is a character-ratio estimate rather than the provider's tokenizer - apply the clamp in `session/llm.ts` after the `chat.params` hook so plugin overrides are checked too, and in the `session/llm/request.ts` twin - leave untouched: configs that already fit, models that budget input separately via `limit.input`, models declaring no window, and windows too small to hold a floor-sized completion (those limits are not credible enough to fail a request on) 11 tests cover the reported case, the floor, and the unchanged paths.
1 parent c59a5a2 commit 1586f59

4 files changed

Lines changed: 340 additions & 3 deletions

File tree

‎packages/opencode/src/provider/transform.ts‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ import type { Provider } from "./provider"
66
import type { ModelsDev } from "./models"
77
import { iife } from "@/util/iife"
88
import { Flag } from "@/flag/flag"
9+
// altimate_change start — output-token clamp needs prompt sizing and a logger
10+
import { Token } from "@/util/token"
11+
import { Log } from "@/util/log"
12+
// altimate_change end
913

1014
type Modality = NonNullable<ModelsDev.Model["modalities"]>["input"][number]
1115

@@ -19,6 +23,9 @@ function mimeToModality(mime: string): Modality | undefined {
1923

2024
export namespace ProviderTransform {
2125
export const OUTPUT_TOKEN_MAX = Flag.OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX || 32_000
26+
// altimate_change start — logger for the output-token clamp below
27+
const log = Log.create({ service: "provider.transform" })
28+
// altimate_change end
2229
// altimate_change start — keep OpenAI encrypted reasoning include values consistent across transforms
2330
const INCLUDE_ENCRYPTED_REASONING = ["reasoning.encrypted_content"] as const
2431

@@ -1278,6 +1285,108 @@ export namespace ProviderTransform {
12781285
// altimate_change end
12791286
}
12801287

1288+
// altimate_change start — clamp the reserved completion budget against the real prompt size.
1289+
//
1290+
// `maxOutputTokens` above is a per-model ceiling. It never reads `limit.context`, so on a
1291+
// model where input and completion share one window a large system prompt can push
1292+
// `input + reservation` past that window and the provider rejects the request with a hard
1293+
// 400 before generating anything. The client already knows all three numbers, so it can
1294+
// either shrink the reservation to fit or refuse with an actionable message.
1295+
1296+
/**
1297+
* Smallest completion budget worth sending. Below this an agent turn cannot reliably emit
1298+
* even one tool call, so a request that would clamp this far is refused instead of being
1299+
* sent to come back as a silently truncated response.
1300+
*/
1301+
export const OUTPUT_TOKEN_FLOOR = 1_024
1302+
1303+
// `inputTokens` is a character-ratio estimate, not the provider's tokenizer. Measured drift
1304+
// against a provider's own accounting on a ~52K prompt was under 1%, so a clamped budget
1305+
// keeps a 2% (minimum 512-token) margin rather than filling the window exactly.
1306+
const CLAMP_MARGIN_FRACTION = 0.02
1307+
const CLAMP_MARGIN_MIN = 512
1308+
1309+
/** Thrown before the request is sent when no usable completion budget fits in the window. */
1310+
export class OutputTokenBudgetError extends Error {
1311+
constructor(
1312+
readonly info: {
1313+
modelID: string
1314+
providerID: string
1315+
inputTokens: number
1316+
requested: number
1317+
context: number
1318+
floor: number
1319+
},
1320+
) {
1321+
super(
1322+
[
1323+
`Context budget exceeded before the request was sent.`,
1324+
`${info.providerID}/${info.modelID} declares a ${info.context}-token context window,`,
1325+
`the prompt is ~${info.inputTokens} tokens, and ${info.requested} tokens are reserved for`,
1326+
`the completion — ${info.inputTokens + info.requested} in total.`,
1327+
`Even after clamping, fewer than ${info.floor} tokens would remain for the response.`,
1328+
`Reduce the system prompt (fewer instructions, skills, or AGENTS.md content), lower the`,
1329+
`output reservation via OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX, or use a model with a`,
1330+
`larger context window.`,
1331+
].join(" "),
1332+
)
1333+
this.name = "OutputTokenBudgetError"
1334+
}
1335+
}
1336+
1337+
/** Rough token count for the prompt about to be sent, used only for the clamp decision. */
1338+
export function estimateInputTokens(system: string[], messages: unknown[]): number {
1339+
return Token.estimate(system.join("\n")) + Token.estimate(JSON.stringify(messages))
1340+
}
1341+
1342+
/**
1343+
* Shrink `requested` so `inputTokens + result` fits `model.limit.context`.
1344+
*
1345+
* Returns `requested` unchanged whenever it already fits, when the caller omitted it, when the
1346+
* model declares no context window, when the declared window is too small to hold even a
1347+
* floor-sized completion (those limits are not credible enough to fail a request on — the
1348+
* provider stays the authority), or when the model budgets input separately via `limit.input`
1349+
* (there the two budgets are not shared and clamping would be wrong).
1350+
* Throws `OutputTokenBudgetError` when the remaining budget is below `OUTPUT_TOKEN_FLOOR`.
1351+
*/
1352+
export function clampOutputTokens(input: {
1353+
model: Provider.Model
1354+
requested: number | undefined
1355+
inputTokens: number
1356+
}): number | undefined {
1357+
const requested = input.requested
1358+
if (requested === undefined) return undefined
1359+
1360+
const context = input.model.limit.context
1361+
if (!context || context <= OUTPUT_TOKEN_FLOOR) return requested
1362+
if (input.model.limit.input) return requested
1363+
if (input.inputTokens <= 0) return requested
1364+
if (input.inputTokens + requested <= context) return requested
1365+
1366+
const margin = Math.max(CLAMP_MARGIN_MIN, Math.ceil(input.inputTokens * CLAMP_MARGIN_FRACTION))
1367+
const clamped = context - input.inputTokens - margin
1368+
if (clamped < OUTPUT_TOKEN_FLOOR) {
1369+
throw new OutputTokenBudgetError({
1370+
modelID: input.model.id,
1371+
providerID: input.model.providerID,
1372+
inputTokens: input.inputTokens,
1373+
requested,
1374+
context,
1375+
floor: OUTPUT_TOKEN_FLOOR,
1376+
})
1377+
}
1378+
log.warn("clamped output token reservation to fit context window", {
1379+
providerID: input.model.providerID,
1380+
modelID: input.model.id,
1381+
context,
1382+
inputTokens: input.inputTokens,
1383+
requested,
1384+
clamped,
1385+
})
1386+
return clamped
1387+
}
1388+
// altimate_change end
1389+
12811390
// altimate_change start — lower MCP/tool JSON Schema to provider-compatible subsets
12821391
type JsonRecord = Record<string, unknown>
12831392

‎packages/opencode/src/session/llm.ts‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,18 @@ export namespace LLM {
150150
)
151151
// altimate_change end
152152

153+
// altimate_change start — clamp the reserved completion budget against the real prompt size.
154+
// `maxOutputTokens` is a per-model ceiling that ignores `limit.context`, so a large system
155+
// prompt could push `input + reservation` past the window and the provider rejected the
156+
// request with a hard 400 before generating anything. Clamped after the chat.params hook so
157+
// a plugin override is checked too. Throws when no usable budget is left.
158+
const maxOutputTokens = ProviderTransform.clampOutputTokens({
159+
model: input.model,
160+
requested: params.maxOutputTokens,
161+
inputTokens: ProviderTransform.estimateInputTokens(system, input.messages),
162+
})
163+
// altimate_change end
164+
153165
const { headers } = await Plugin.trigger(
154166
"chat.headers",
155167
{
@@ -249,8 +261,9 @@ export namespace LLM {
249261
activeTools: Object.keys(tools).filter((x) => x !== "invalid"),
250262
tools,
251263
toolChoice: input.toolChoice,
252-
// altimate_change start — read maxOutputTokens from params (now plumbed through chat.params hook)
253-
maxOutputTokens: params.maxOutputTokens,
264+
// altimate_change start — read maxOutputTokens from params (now plumbed through chat.params
265+
// hook), clamped above so it cannot exceed the model's context window
266+
maxOutputTokens,
254267
// altimate_change end
255268
abortSignal: input.abort,
256269
headers: {

‎packages/opencode/src/session/llm/request.ts‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,11 +168,27 @@ export const prepare = Effect.fn("LLMRequestPrep.prepare")(function* (input: Pre
168168
? (yield* InstanceState.context).project.id
169169
: undefined
170170

171+
// altimate_change start — clamp the reserved completion budget against the real prompt size.
172+
// `maxOutputTokens` is a per-model ceiling that ignores `limit.context`, so a large system
173+
// prompt could push `input + reservation` past the window and the provider rejected the request
174+
// with a hard 400 before generating anything. `input.messages` is used rather than the merged
175+
// `messages` because the latter can already carry `system` as leading system messages, which
176+
// would double-count the prompt. Throws when no usable budget is left.
177+
const clampedParams = {
178+
...params,
179+
maxOutputTokens: ProviderTransform.clampOutputTokens({
180+
model: input.model,
181+
requested: params.maxOutputTokens,
182+
inputTokens: ProviderTransform.estimateInputTokens(system, input.messages),
183+
}),
184+
}
185+
// altimate_change end
186+
171187
return {
172188
system,
173189
messages,
174190
tools: Object.fromEntries(Object.entries(tools).toSorted(([a], [b]) => a.localeCompare(b))),
175-
params,
191+
params: clampedParams,
176192
messageTransformOptions: options,
177193
headers: {
178194
...(input.model.providerID.startsWith("opencode")

‎packages/opencode/test/provider/transform.test.ts‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { describe, expect, test } from "bun:test"
22
import { Effect } from "effect"
3+
import type { ModelMessage } from "ai"
34
import { ProviderTransform } from "@/provider/transform"
45
import { LLMRequestPrep } from "@/session/llm/request"
56
// ProviderTransform.message expects a Provider.Model with the fork's ModelID/ProviderID brands.
@@ -4650,3 +4651,201 @@ describe("ProviderTransform.providerOptions - ai-gateway-provider", () => {
46504651
expect(result).toEqual({ openaiCompatible: { reasoningEffort: "high" } })
46514652
})
46524653
})
4654+
4655+
// Regression suite for the output-token reservation exceeding the context window.
4656+
//
4657+
// Reported failure: with a large system prompt on a model declaring a 65,536-token window, the
4658+
// shipped 16,384-token reservation produced a hard provider 400 before any model work —
4659+
// "You requested a total of 68564 tokens: 52180 tokens from the input messages and 16384 tokens
4660+
// for the completion". The reservation was applied without ever being compared to the input size.
4661+
describe("ProviderTransform.clampOutputTokens", () => {
4662+
const createWindowModel = (limit: { context: number; input?: number; output: number }) =>
4663+
({
4664+
id: "large-window-model",
4665+
providerID: "openai-compatible",
4666+
api: { id: "large-window-model", url: "https://example.invalid/v1", npm: "@ai-sdk/openai-compatible" },
4667+
name: "Large Window Model",
4668+
capabilities: {
4669+
temperature: true,
4670+
reasoning: false,
4671+
attachment: false,
4672+
toolcall: true,
4673+
input: { text: true, audio: false, image: false, video: false, pdf: false },
4674+
output: { text: true, audio: false, image: false, video: false, pdf: false },
4675+
interleaved: false,
4676+
},
4677+
cost: { input: 1, output: 1, cache: { read: 0, write: 0 } },
4678+
limit,
4679+
status: "active",
4680+
options: {},
4681+
headers: {},
4682+
}) as any
4683+
4684+
// The exact reported case.
4685+
const REPORTED = { inputTokens: 52_180, requested: 16_384, context: 65_536 }
4686+
4687+
test("the reported case is clamped instead of being sent as-is", () => {
4688+
const model = createWindowModel({ context: REPORTED.context, output: 16_384 })
4689+
const result = ProviderTransform.clampOutputTokens({
4690+
model,
4691+
requested: REPORTED.requested,
4692+
inputTokens: REPORTED.inputTokens,
4693+
})!
4694+
4695+
// Unclamped, this is exactly the request the provider rejected.
4696+
expect(REPORTED.inputTokens + REPORTED.requested).toBeGreaterThan(REPORTED.context)
4697+
expect(result).toBeLessThan(REPORTED.requested)
4698+
// The clamped request fits, with the estimator margin (2%, min 512) still spare.
4699+
expect(REPORTED.inputTokens + result).toBeLessThanOrEqual(REPORTED.context)
4700+
// 65536 - 52180 - ceil(52180 * 0.02) = 12312
4701+
expect(result).toBe(12_312)
4702+
// Still a usable completion budget, not a stub.
4703+
expect(result).toBeGreaterThanOrEqual(ProviderTransform.OUTPUT_TOKEN_FLOOR)
4704+
})
4705+
4706+
test("throws with the actual numbers when even the floor does not fit", () => {
4707+
const model = createWindowModel({ context: 65_536, output: 16_384 })
4708+
let thrown: unknown
4709+
try {
4710+
ProviderTransform.clampOutputTokens({ model, requested: 16_384, inputTokens: 65_000 })
4711+
} catch (e) {
4712+
thrown = e
4713+
}
4714+
expect(thrown).toBeInstanceOf(ProviderTransform.OutputTokenBudgetError)
4715+
const message = (thrown as Error).message
4716+
// The message must name input tokens, the requested reservation and the window.
4717+
expect(message).toContain("65000")
4718+
expect(message).toContain("16384")
4719+
expect(message).toContain("65536")
4720+
expect(message).toContain("OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX")
4721+
})
4722+
4723+
test("throws rather than clamping to an unusable budget just above zero", () => {
4724+
const model = createWindowModel({ context: 65_536, output: 16_384 })
4725+
// 65536 - 65000 = 536 would "fit" arithmetically but is below the floor.
4726+
expect(536).toBeLessThan(ProviderTransform.OUTPUT_TOKEN_FLOOR)
4727+
expect(() => ProviderTransform.clampOutputTokens({ model, requested: 16_384, inputTokens: 65_000 })).toThrow()
4728+
})
4729+
4730+
test("leaves a config that already fits completely unchanged", () => {
4731+
const model = createWindowModel({ context: 65_536, output: 16_384 })
4732+
expect(ProviderTransform.clampOutputTokens({ model, requested: 16_384, inputTokens: 10_000 })).toBe(16_384)
4733+
// Exactly filling the window is still a valid request and must not be touched.
4734+
expect(ProviderTransform.clampOutputTokens({ model, requested: 16_384, inputTokens: 49_152 })).toBe(16_384)
4735+
})
4736+
4737+
test("leaves large-window models unchanged", () => {
4738+
const model = createWindowModel({ context: 200_000, output: 8_192 })
4739+
expect(ProviderTransform.clampOutputTokens({ model, requested: 8_192, inputTokens: 150_000 })).toBe(8_192)
4740+
})
4741+
4742+
test("does not clamp models that budget input separately from output", () => {
4743+
// limit.input means the two budgets are not shared, so input + output can exceed context.
4744+
const model = createWindowModel({ context: 65_536, input: 65_536, output: 16_384 })
4745+
expect(ProviderTransform.clampOutputTokens({ model, requested: 16_384, inputTokens: 60_000 })).toBe(16_384)
4746+
})
4747+
4748+
test("does not clamp when the model declares no context window", () => {
4749+
const model = createWindowModel({ context: 0, output: 16_384 })
4750+
expect(ProviderTransform.clampOutputTokens({ model, requested: 16_384, inputTokens: 60_000 })).toBe(16_384)
4751+
})
4752+
4753+
test("does not clamp a window too small to hold even a floor-sized completion", () => {
4754+
// A declared window this small is a placeholder or a test fixture, not a real limit. Failing
4755+
// the request client-side on numbers we do not believe would be worse than letting the
4756+
// provider answer, so the guard stays out of the way.
4757+
const model = createWindowModel({ context: 20, output: 10 })
4758+
expect(ProviderTransform.clampOutputTokens({ model, requested: 10, inputTokens: 11 })).toBe(10)
4759+
})
4760+
4761+
test("passes an omitted reservation through untouched", () => {
4762+
// Codex and GitHub Copilot deliberately send no maxOutputTokens.
4763+
const model = createWindowModel({ context: 65_536, output: 16_384 })
4764+
expect(ProviderTransform.clampOutputTokens({ model, requested: undefined, inputTokens: 60_000 })).toBeUndefined()
4765+
})
4766+
})
4767+
4768+
describe("LLMRequestPrep.prepare - output token reservation", () => {
4769+
const sessionID = "ses_clamp-test"
4770+
4771+
const model = {
4772+
id: "large-window-model",
4773+
providerID: "openai-compatible",
4774+
api: { id: "large-window-model", url: "https://example.invalid/v1", npm: "@ai-sdk/openai-compatible" },
4775+
name: "Large Window Model",
4776+
capabilities: {
4777+
temperature: true,
4778+
reasoning: false,
4779+
attachment: false,
4780+
toolcall: true,
4781+
input: { text: true, audio: false, image: false, video: false, pdf: false },
4782+
output: { text: true, audio: false, image: false, video: false, pdf: false },
4783+
interleaved: false,
4784+
},
4785+
cost: { input: 1, output: 1, cache: { read: 0, write: 0 } },
4786+
limit: { context: 65_536, output: 16_384 },
4787+
status: "active",
4788+
options: {},
4789+
headers: {},
4790+
} as any
4791+
4792+
const messages: ModelMessage[] = [{ role: "user", content: "Hello" }]
4793+
4794+
const run = (systemPrompt: string) =>
4795+
Effect.runPromise(
4796+
LLMRequestPrep.prepare({
4797+
user: {
4798+
id: "msg_user-clamp",
4799+
sessionID,
4800+
role: "user",
4801+
time: { created: Date.now() },
4802+
agent: "test",
4803+
model: { providerID: "openai-compatible", modelID: "large-window-model" },
4804+
} as any,
4805+
sessionID,
4806+
model,
4807+
agent: { name: "test", mode: "primary", options: {}, permission: [], prompt: systemPrompt } as any,
4808+
system: [],
4809+
messages,
4810+
tools: {},
4811+
provider: { id: "openai-compatible", options: {} } as any,
4812+
auth: undefined,
4813+
plugin: {
4814+
trigger: (_name: string, _input: unknown, output: unknown) => Effect.succeed(output),
4815+
list: () => Effect.succeed([]),
4816+
init: () => Effect.void,
4817+
} as any,
4818+
flags: { outputTokenMax: 32_000, client: "test" } as any,
4819+
isWorkflow: false,
4820+
}),
4821+
)
4822+
4823+
// Prose with no code or JSON characters, so Token.estimate uses its default 3.7 chars/token.
4824+
const PROSE = "the quick brown fox jumps over the lazy dog "
4825+
const largePrompt = PROSE.repeat(Math.ceil((52_180 * 3.7) / PROSE.length))
4826+
4827+
test("a ~52K-token system prompt does not produce an unclamped request", async () => {
4828+
const estimated = ProviderTransform.estimateInputTokens([largePrompt], messages)
4829+
// Sized to reproduce the reported 52,180-token prompt.
4830+
expect(estimated).toBeGreaterThan(51_500)
4831+
expect(estimated).toBeLessThan(53_500)
4832+
// Unclamped this is the request the provider rejected: 52,180 + 16,384 > 65,536.
4833+
expect(estimated + 16_384).toBeGreaterThan(65_536)
4834+
4835+
const result = await run(largePrompt)
4836+
const maxOutputTokens = result.params.maxOutputTokens!
4837+
expect(maxOutputTokens).toBeLessThan(16_384)
4838+
expect(estimated + maxOutputTokens).toBeLessThanOrEqual(65_536)
4839+
expect(maxOutputTokens).toBeGreaterThanOrEqual(ProviderTransform.OUTPUT_TOKEN_FLOOR)
4840+
})
4841+
4842+
test("a small system prompt keeps the full model reservation", async () => {
4843+
const result = await run("You are a helpful assistant.")
4844+
expect(result.params.maxOutputTokens).toBe(16_384)
4845+
})
4846+
4847+
test("a system prompt that leaves no usable budget fails before the request is built", async () => {
4848+
const hugePrompt = PROSE.repeat(Math.ceil((65_000 * 3.7) / PROSE.length))
4849+
await expect(run(hugePrompt)).rejects.toThrow(/Context budget exceeded/)
4850+
})
4851+
})

0 commit comments

Comments
 (0)