Skip to content

Commit b06207a

Browse files
anandgupta42claude
andcommitted
fix: count what is actually sent when clamping the output reservation
Review of the clamp found five ways the guard read the wrong numbers. Two were regressions the clamp itself introduced, so they are fixed here rather than deferred. - count tool definitions in the estimate. Providers bill tool schemas as prompt tokens, so omitting them under-counted exactly the tool-heavy sessions most likely to overflow. In `session/llm.ts` the clamp moves below `resolveTools` and the retrieval filter — still after the `chat.params` hook — so it sees the tools actually sent; `session/llm/request.ts` already had them in scope. - strip media before estimating. Attachments arrive as base64 payloads or byte arrays on the model messages, and `JSON.stringify` counted a 1 MiB screenshot as hundreds of thousands of text tokens, refusing a multimodal request the provider would have accepted. `Compaction.estimate` already does this via `stripMedia`; this follows the same trade-off. - apply the estimator margin to the fit check, not just to the clamp. The boundary was discontinuous: one token over clamped to a ~1K cushion, one token under was sent with no protection at all, even though the input count is a character-ratio estimate. - clamp models that declare `limit.input`. That field is an input ceiling inside the shared window, not a separate budget: `Session.Overflow.usable` subtracts the reserved completion from it, and catalog-shaped fixtures pair it with an equal or larger `context` (context 200K / input 200K / output 32K). The early return left the exact 400 this guard prevents. - clamp the configured reasoning budget alongside `maxOutputTokens`. Anthropic rejects a request whose thinking budget is not below `max_tokens`, and this repository configures fixed 16,000/31,999-token budgets, so a clamp on its own turned one provider 400 into another. The floor also rises when a reasoning variant is configured so thinking and an answer both fit. Also honours a context window widened by a request header — the GitLab AI-gateway loader always sends `anthropic-beta: context-1m-2025-08-07` while its catalog entries still declare 200K — and takes the prompt estimate lazily so the history is not serialized on paths that return before it is needed. 13 new tests; the two that pinned the old exact-fill and `limit.input` behaviour are rewritten to assert the corrected outcome. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ
1 parent 8c289cb commit b06207a

4 files changed

Lines changed: 314 additions & 47 deletions

File tree

‎packages/opencode/src/provider/transform.ts‎

Lines changed: 125 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1334,52 +1334,155 @@ export namespace ProviderTransform {
13341334
}
13351335
}
13361336

1337-
/** Rough token count for the prompt about to be sent, used only for the clamp decision. */
1338-
export function estimateInputTokens(system: string[], messages: unknown[]): number {
1339-
return Token.estimate(system.join("\n")) + Token.estimate(JSON.stringify(messages))
1337+
/**
1338+
* Smallest reasoning budget the Anthropic API accepts. When a reasoning variant is configured
1339+
* the clamp has to leave room for both the thinking budget and an actual answer, so the floor
1340+
* rises by this much.
1341+
*/
1342+
const MIN_REASONING_BUDGET = 1_024
1343+
1344+
// Attachments reach this point as base64 payloads or byte arrays on the model messages.
1345+
// Serializing them whole would count a 1 MiB screenshot as hundreds of thousands of text
1346+
// tokens and refuse a request the provider would have accepted, so they are replaced with a
1347+
// marker before estimating — the same trade-off `Compaction.estimate` makes via `stripMedia`.
1348+
const MEDIA_PLACEHOLDER = "[media]"
1349+
1350+
function stringifyWithoutMedia(value: unknown): string {
1351+
return JSON.stringify(value, (key, val) => {
1352+
if (ArrayBuffer.isView(val) || val instanceof ArrayBuffer) return MEDIA_PLACEHOLDER
1353+
if (typeof val !== "string") return val
1354+
if (key === "data" || key === "image") return MEDIA_PLACEHOLDER
1355+
if (key === "url" && val.startsWith("data:")) return MEDIA_PLACEHOLDER
1356+
return val
1357+
})
1358+
}
1359+
1360+
/**
1361+
* Rough token count for the prompt about to be sent, used only for the clamp decision.
1362+
*
1363+
* Counts the system text, the messages with media stripped, and the tool definitions — tools
1364+
* are sent with the request and providers bill them as prompt tokens, so leaving them out
1365+
* would under-count exactly the tool-heavy sessions most likely to overflow.
1366+
*/
1367+
export function estimateInputTokens(system: string[], messages: unknown[], tools?: unknown): number {
1368+
return (
1369+
Token.estimate(system.join("\n")) +
1370+
Token.estimate(stringifyWithoutMedia(messages)) +
1371+
(tools ? Token.estimate(stringifyWithoutMedia(tools)) : 0)
1372+
)
1373+
}
1374+
1375+
/**
1376+
* The context window actually in force for a request.
1377+
*
1378+
* The catalog value is a static declaration, but a request header can enable a larger window:
1379+
* the GitLab AI-gateway loader always sends `anthropic-beta: context-1m-2025-08-07` while its
1380+
* catalog entries still declare 200K. Clamping against the catalog number there would shrink
1381+
* or refuse prompts the provider route accepts.
1382+
*/
1383+
export function effectiveContext(model: Provider.Model, headers?: Record<string, string>): number {
1384+
const context = model.limit.context
1385+
if (headers?.["anthropic-beta"]?.includes("context-1m-2025-08-07")) return Math.max(context, 1_000_000)
1386+
return context
1387+
}
1388+
1389+
/** Largest reasoning budget configured anywhere in the provider options, or 0 if none. */
1390+
export function configuredReasoningBudget(options: unknown): number {
1391+
let budget = 0
1392+
const visit = (value: unknown) => {
1393+
if (Array.isArray(value)) return value.forEach(visit)
1394+
if (typeof value !== "object" || value === null) return
1395+
for (const [key, val] of Object.entries(value)) {
1396+
if ((key === "budgetTokens" || key === "thinkingBudget") && typeof val === "number") {
1397+
budget = Math.max(budget, val)
1398+
} else visit(val)
1399+
}
1400+
}
1401+
visit(options)
1402+
return budget
1403+
}
1404+
1405+
/**
1406+
* Lower a configured reasoning budget so it stays below the (possibly clamped) completion
1407+
* budget. Anthropic rejects a request whose `thinking.budget_tokens` is not less than
1408+
* `max_tokens`, and this repository configures fixed 16,000/31,999-token budgets, so clamping
1409+
* `maxOutputTokens` on its own can turn one provider 400 into another.
1410+
*/
1411+
export function clampReasoningBudget<T>(options: T, maxOutputTokens: number | undefined): T {
1412+
if (maxOutputTokens === undefined) return options
1413+
const ceiling = maxOutputTokens - OUTPUT_TOKEN_FLOOR
1414+
if (ceiling < MIN_REASONING_BUDGET) return options
1415+
1416+
let changed = false
1417+
const next = JSON.parse(JSON.stringify(options ?? null), (key, val) => {
1418+
if ((key === "budgetTokens" || key === "thinkingBudget") && typeof val === "number" && val > ceiling) {
1419+
changed = true
1420+
return ceiling
1421+
}
1422+
return val
1423+
})
1424+
return changed ? next : options
13401425
}
13411426

13421427
/**
1343-
* Shrink `requested` so `inputTokens + result` fits `model.limit.context`.
1428+
* Shrink `requested` so `inputTokens + result` fits the context window.
1429+
*
1430+
* Returns `requested` unchanged whenever it already fits with the estimator margin to spare,
1431+
* when the caller omitted it, when the model declares no context window, or when the declared
1432+
* window is too small to hold even a floor-sized completion (those limits are not credible
1433+
* enough to fail a request on — the provider stays the authority).
1434+
* Throws `OutputTokenBudgetError` when the remaining budget is below the floor.
13441435
*
1345-
* Returns `requested` unchanged whenever it already fits, when the caller omitted it, when the
1346-
* model declares no context window, when the declared window is too small to hold even a
1347-
* floor-sized completion (those limits are not credible enough to fail a request on — the
1348-
* provider stays the authority), or when the model budgets input separately via `limit.input`
1349-
* (there the two budgets are not shared and clamping would be wrong).
1350-
* Throws `OutputTokenBudgetError` when the remaining budget is below `OUTPUT_TOKEN_FLOOR`.
1436+
* `inputTokens` may be a thunk: serializing the whole history is wasted work on the paths that
1437+
* return before the estimate is needed (Codex and GitHub Copilot send no reservation at all).
13511438
*/
13521439
export function clampOutputTokens(input: {
13531440
model: Provider.Model
13541441
requested: number | undefined
1355-
inputTokens: number
1442+
inputTokens: number | (() => number)
1443+
/** Overrides `model.limit.context`; see `effectiveContext`. */
1444+
context?: number
1445+
/** Configured reasoning budget, if any — raises the floor so thinking still fits. */
1446+
reasoningBudget?: number
13561447
}): number | undefined {
13571448
const requested = input.requested
13581449
if (requested === undefined) return undefined
13591450

1360-
const context = input.model.limit.context
1361-
if (!context || context <= OUTPUT_TOKEN_FLOOR) return requested
1362-
if (input.model.limit.input) return requested
1363-
if (input.inputTokens <= 0) return requested
1364-
if (input.inputTokens + requested <= context) return requested
1451+
// A configured reasoning budget has to fit under the clamped value alongside a real answer.
1452+
const floor =
1453+
input.reasoningBudget && input.reasoningBudget > 0
1454+
? OUTPUT_TOKEN_FLOOR + MIN_REASONING_BUDGET
1455+
: OUTPUT_TOKEN_FLOOR
1456+
1457+
const context = input.context ?? input.model.limit.context
1458+
if (!context || context <= floor) return requested
1459+
1460+
const inputTokens = typeof input.inputTokens === "function" ? input.inputTokens() : input.inputTokens
1461+
if (inputTokens <= 0) return requested
1462+
1463+
// The margin applies to the fit check as well as the clamp. `inputTokens` is a character-ratio
1464+
// estimate, so a prompt that appears to fill the window exactly can still overflow it; without
1465+
// this the boundary is discontinuous — one token over clamps to a ~1K cushion, one token under
1466+
// gets no protection at all.
1467+
const margin = Math.max(CLAMP_MARGIN_MIN, Math.ceil(inputTokens * CLAMP_MARGIN_FRACTION))
1468+
if (inputTokens + requested + margin <= context) return requested
13651469

1366-
const margin = Math.max(CLAMP_MARGIN_MIN, Math.ceil(input.inputTokens * CLAMP_MARGIN_FRACTION))
1367-
const clamped = context - input.inputTokens - margin
1368-
if (clamped < OUTPUT_TOKEN_FLOOR) {
1470+
const clamped = context - inputTokens - margin
1471+
if (clamped < floor) {
13691472
throw new OutputTokenBudgetError({
13701473
modelID: input.model.id,
13711474
providerID: input.model.providerID,
1372-
inputTokens: input.inputTokens,
1475+
inputTokens,
13731476
requested,
13741477
context,
1375-
floor: OUTPUT_TOKEN_FLOOR,
1478+
floor,
13761479
})
13771480
}
13781481
log.warn("clamped output token reservation to fit context window", {
13791482
providerID: input.model.providerID,
13801483
modelID: input.model.id,
13811484
context,
1382-
inputTokens: input.inputTokens,
1485+
inputTokens,
13831486
requested,
13841487
clamped,
13851488
})

‎packages/opencode/src/session/llm.ts‎

Lines changed: 24 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -150,18 +150,6 @@ export namespace LLM {
150150
)
151151
// altimate_change end
152152

153-
// altimate_change start — clamp the reserved completion budget against the real prompt size.
154-
// `maxOutputTokens` is a per-model ceiling that ignores `limit.context`, so a large system
155-
// prompt could push `input + reservation` past the window and the provider rejected the
156-
// request with a hard 400 before generating anything. Clamped after the chat.params hook so
157-
// a plugin override is checked too. Throws when no usable budget is left.
158-
const maxOutputTokens = ProviderTransform.clampOutputTokens({
159-
model: input.model,
160-
requested: params.maxOutputTokens,
161-
inputTokens: ProviderTransform.estimateInputTokens(system, input.messages),
162-
})
163-
// altimate_change end
164-
165153
const { headers } = await Plugin.trigger(
166154
"chat.headers",
167155
{
@@ -223,6 +211,27 @@ export namespace LLM {
223211
}
224212
// altimate_change end
225213

214+
// altimate_change start — clamp the reserved completion budget against the real prompt size.
215+
// `maxOutputTokens` is a per-model ceiling that ignores `limit.context`, so a large system
216+
// prompt could push `input + reservation` past the window and the provider rejected the
217+
// request with a hard 400 before generating anything. Placed after the chat.params hook so a
218+
// plugin override is checked too, and after tool resolution and retrieval filtering so the
219+
// estimate covers the tool schemas actually sent. Throws when no usable budget is left.
220+
// Read outside the closure below: `system` is declared as an untyped array upstream and its
221+
// evolved element type is not visible from inside a callback.
222+
const systemParts: string[] = system
223+
const reasoningBudget = ProviderTransform.configuredReasoningBudget(params.options)
224+
const maxOutputTokens = ProviderTransform.clampOutputTokens({
225+
model: input.model,
226+
requested: params.maxOutputTokens,
227+
context: ProviderTransform.effectiveContext(input.model, headers),
228+
reasoningBudget,
229+
inputTokens: () => ProviderTransform.estimateInputTokens(systemParts, input.messages, tools),
230+
})
231+
// A clamped completion budget must still leave room for the configured thinking budget.
232+
const clampedOptions = ProviderTransform.clampReasoningBudget(params.options, maxOutputTokens)
233+
// altimate_change end
234+
226235
return streamText({
227236
onError(error) {
228237
l.error("stream error", {
@@ -257,7 +266,9 @@ export namespace LLM {
257266
temperature: params.temperature,
258267
topP: params.topP,
259268
topK: params.topK,
260-
providerOptions: ProviderTransform.providerOptions(input.model, params.options),
269+
// altimate_change start — use the reasoning-budget-clamped options built above
270+
providerOptions: ProviderTransform.providerOptions(input.model, clampedOptions),
271+
// altimate_change end
261272
activeTools: Object.keys(tools).filter((x) => x !== "invalid"),
262273
tools,
263274
toolChoice: input.toolChoice,

‎packages/opencode/src/session/llm/request.ts‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -173,14 +173,20 @@ export const prepare = Effect.fn("LLMRequestPrep.prepare")(function* (input: Pre
173173
// prompt could push `input + reservation` past the window and the provider rejected the request
174174
// with a hard 400 before generating anything. `input.messages` is used rather than the merged
175175
// `messages` because the latter can already carry `system` as leading system messages, which
176-
// would double-count the prompt. Throws when no usable budget is left.
176+
// would double-count the prompt. Tool schemas are counted too — providers bill them as prompt
177+
// tokens. Throws when no usable budget is left.
178+
const clampedMaxOutputTokens = ProviderTransform.clampOutputTokens({
179+
model: input.model,
180+
requested: params.maxOutputTokens,
181+
context: ProviderTransform.effectiveContext(input.model, headers),
182+
reasoningBudget: ProviderTransform.configuredReasoningBudget(params.options),
183+
inputTokens: () => ProviderTransform.estimateInputTokens(system, input.messages, tools),
184+
})
177185
const clampedParams = {
178186
...params,
179-
maxOutputTokens: ProviderTransform.clampOutputTokens({
180-
model: input.model,
181-
requested: params.maxOutputTokens,
182-
inputTokens: ProviderTransform.estimateInputTokens(system, input.messages),
183-
}),
187+
maxOutputTokens: clampedMaxOutputTokens,
188+
// A clamped completion budget must still leave room for the configured thinking budget.
189+
options: ProviderTransform.clampReasoningBudget(params.options, clampedMaxOutputTokens),
184190
}
185191
// altimate_change end
186192

0 commit comments

Comments
 (0)