Skip to content

Commit 38af0dc

Browse files
committed
refactor(llm): centralize codec scaffolding, ToolAccumulator, and totalTokens policy
Three review findings collapsed into one ProviderShared pass. M1: Five adapters duplicated the same six-line block: const ChunkJson = Schema.fromJsonString(Chunk) const TargetJson = Schema.fromJsonString(Target) const decodeChunkSync = Schema.decodeUnknownSync(ChunkJson) const encodeTarget = Schema.encodeSync(TargetJson) const decodeTarget = Schema.decodeUnknownEffect(Draft.pipe(Schema.decodeTo(Target))) const decodeChunk = (data) => Effect.try({...chunkError(...)}) Lift it into `ProviderShared.codecs({ adapter, draft, target, chunk, chunkErrorMessage })` returning `{ encodeTarget, decodeTarget, decodeChunk }`. The result drops directly into `Adapter.define`'s `validate` field (uses `validateWith` internally to map parse errors to InvalidRequestError). Adopted in OpenAI Chat, OpenAI Responses, Anthropic Messages, and Gemini. Bedrock has a custom event-stream `decodeChunk` that takes `unknown` (not `string`) so it keeps its inline codecs. M2: Four adapters defined an identical `ToolAccumulator` interface (`{ readonly id: string; readonly name: string; readonly input: string }`). Lift to `ProviderShared.ToolAccumulator`. Anthropic extends it locally with `providerExecuted` for hosted tools. M3: The five `mapUsage` implementations had subtly different `totalTokens` policies — OpenAI Chat passed through whatever the provider sent, OpenAI Responses unconditionally summed inputs and output (publishing `totalTokens: 0` when both were `undefined`), Anthropic and Gemini guarded with conditionals, Bedrock used a `(...) || undefined` falsy fallback. Add `ProviderShared.totalTokens` with one rule: prefer provider-supplied total, else sum inputs and outputs only when at least one is defined, else `undefined`. Fixes the OpenAI Responses `totalTokens: 0` bug. M6: Anthropic's `mergeUsage` recomputed `totalTokens` from the merged input/output via two nested ?? chains and a conditional sum. Simplified to use the same totalTokens helper, with `inputTokens` and `outputTokens` extracted as locals so the merge is one ?? per field and the comment explains why merging exists (Anthropic emits usage on `message_start` and `message_delta`). No behavior changes other than the OpenAI Responses fix; existing tests pass unchanged. 120 LLM-package tests + 33 OpenCode bridge tests green.
1 parent 8bbbcee commit 38af0dc

5 files changed

Lines changed: 110 additions & 78 deletions

File tree

‎packages/llm/src/provider/anthropic-messages.ts‎

Lines changed: 18 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -179,10 +179,7 @@ const AnthropicChunk = Schema.Struct({
179179
})
180180
type AnthropicChunk = Schema.Schema.Type<typeof AnthropicChunk>
181181

182-
interface ToolAccumulator {
183-
readonly id: string
184-
readonly name: string
185-
readonly input: string
182+
interface ToolAccumulator extends ProviderShared.ToolAccumulator {
186183
readonly providerExecuted: boolean
187184
}
188185

@@ -191,17 +188,13 @@ interface ParserState {
191188
readonly usage?: Usage
192189
}
193190

194-
const AnthropicChunkJson = Schema.fromJsonString(AnthropicChunk)
195-
const AnthropicTargetJson = Schema.fromJsonString(AnthropicMessagesTarget)
196-
const decodeChunkSync = Schema.decodeUnknownSync(AnthropicChunkJson)
197-
198-
const decodeChunk = (data: string) =>
199-
Effect.try({
200-
try: () => decodeChunkSync(data),
201-
catch: () => ProviderShared.chunkError(ADAPTER, "Invalid Anthropic Messages stream chunk", data),
202-
})
203-
const encodeTarget = Schema.encodeSync(AnthropicTargetJson)
204-
const decodeTarget = Schema.decodeUnknownEffect(AnthropicMessagesDraft.pipe(Schema.decodeTo(AnthropicMessagesTarget)))
191+
const { encodeTarget, decodeTarget, decodeChunk } = ProviderShared.codecs({
192+
adapter: ADAPTER,
193+
draft: AnthropicMessagesDraft,
194+
target: AnthropicMessagesTarget,
195+
chunk: AnthropicChunk,
196+
chunkErrorMessage: "Invalid Anthropic Messages stream chunk",
197+
})
205198

206199
const invalid = ProviderShared.invalidRequest
207200

@@ -363,24 +356,26 @@ const mapUsage = (usage: AnthropicUsage | undefined): Usage | undefined => {
363356
outputTokens: usage.output_tokens,
364357
cacheReadInputTokens: usage.cache_read_input_tokens ?? undefined,
365358
cacheWriteInputTokens: usage.cache_creation_input_tokens ?? undefined,
366-
totalTokens: usage.input_tokens !== undefined || usage.output_tokens !== undefined
367-
? (usage.input_tokens ?? 0) + (usage.output_tokens ?? 0)
368-
: undefined,
359+
totalTokens: ProviderShared.totalTokens(usage.input_tokens, usage.output_tokens, undefined),
369360
native: usage,
370361
})
371362
}
372363

364+
// Anthropic emits usage on `message_start` and again on `message_delta` — the
365+
// final delta carries the authoritative totals. Right-biased merge: each
366+
// field prefers `right` when defined, falls back to `left`. `totalTokens` is
367+
// recomputed from the merged input/output to stay consistent.
373368
const mergeUsage = (left: Usage | undefined, right: Usage | undefined) => {
374369
if (!left) return right
375370
if (!right) return left
371+
const inputTokens = right.inputTokens ?? left.inputTokens
372+
const outputTokens = right.outputTokens ?? left.outputTokens
376373
return new Usage({
377-
inputTokens: right.inputTokens ?? left.inputTokens,
378-
outputTokens: right.outputTokens ?? left.outputTokens,
374+
inputTokens,
375+
outputTokens,
379376
cacheReadInputTokens: right.cacheReadInputTokens ?? left.cacheReadInputTokens,
380377
cacheWriteInputTokens: right.cacheWriteInputTokens ?? left.cacheWriteInputTokens,
381-
totalTokens: (right.inputTokens ?? left.inputTokens) !== undefined || (right.outputTokens ?? left.outputTokens) !== undefined
382-
? (right.inputTokens ?? left.inputTokens ?? 0) + (right.outputTokens ?? left.outputTokens ?? 0)
383-
: undefined,
378+
totalTokens: ProviderShared.totalTokens(inputTokens, outputTokens, undefined),
384379
native: { ...left.native, ...right.native },
385380
})
386381
}

‎packages/llm/src/provider/bedrock-converse.ts‎

Lines changed: 3 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -592,31 +592,23 @@ const mapUsage = (usage: BedrockUsageSchema | undefined): Usage | undefined => {
592592
return new Usage({
593593
inputTokens: usage.inputTokens,
594594
outputTokens: usage.outputTokens,
595-
totalTokens:
596-
usage.totalTokens ??
597-
((usage.inputTokens ?? 0) + (usage.outputTokens ?? 0) || undefined),
595+
totalTokens: ProviderShared.totalTokens(usage.inputTokens, usage.outputTokens, usage.totalTokens),
598596
cacheReadInputTokens: usage.cacheReadInputTokens,
599597
cacheWriteInputTokens: usage.cacheWriteInputTokens,
600598
native: usage,
601599
})
602600
}
603601

604-
interface ToolAccumulator {
605-
readonly id: string
606-
readonly name: string
607-
readonly input: string
608-
}
609-
610602
interface ParserState {
611-
readonly tools: Record<number, ToolAccumulator>
603+
readonly tools: Record<number, ProviderShared.ToolAccumulator>
612604
// Bedrock splits the finish into `messageStop` (carries `stopReason`) and
613605
// `metadata` (carries usage). The raw stop reason is held here until
614606
// `metadata` arrives, then mapped + emitted together as a single terminal
615607
// `request-finish` event so consumers see one event with both.
616608
readonly pendingStopReason: string | undefined
617609
}
618610

619-
const finishToolCall = (tool: ToolAccumulator | undefined) =>
611+
const finishToolCall = (tool: ProviderShared.ToolAccumulator | undefined) =>
620612
Effect.gen(function* () {
621613
if (!tool) return [] as ReadonlyArray<LLMEvent>
622614
const input = yield* ProviderShared.parseToolInput(ADAPTER, tool.name, tool.input)

‎packages/llm/src/provider/openai-chat.ts‎

Lines changed: 11 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -129,22 +129,13 @@ const OpenAIChatChunk = Schema.Struct({
129129
})
130130
type OpenAIChatChunk = Schema.Schema.Type<typeof OpenAIChatChunk>
131131

132-
const OpenAIChatChunkJson = Schema.fromJsonString(OpenAIChatChunk)
133-
const OpenAIChatTargetJson = Schema.fromJsonString(OpenAIChatTarget)
134-
const decodeChunkSync = Schema.decodeUnknownSync(OpenAIChatChunkJson)
135-
const encodeTarget = Schema.encodeSync(OpenAIChatTargetJson)
136-
137-
const decodeChunk = (data: string) =>
138-
Effect.try({
139-
try: () => decodeChunkSync(data),
140-
catch: () => ProviderShared.chunkError(ADAPTER, "Invalid OpenAI Chat stream chunk", data),
141-
})
142-
143-
interface ToolAccumulator {
144-
readonly id: string
145-
readonly name: string
146-
readonly input: string
147-
}
132+
const { encodeTarget, decodeTarget, decodeChunk } = ProviderShared.codecs({
133+
adapter: ADAPTER,
134+
draft: OpenAIChatDraft,
135+
target: OpenAIChatTarget,
136+
chunk: OpenAIChatChunk,
137+
chunkErrorMessage: "Invalid OpenAI Chat stream chunk",
138+
})
148139

149140
interface ParsedToolCall {
150141
readonly id: string
@@ -153,14 +144,12 @@ interface ParsedToolCall {
153144
}
154145

155146
interface ParserState {
156-
readonly tools: Record<number, ToolAccumulator>
147+
readonly tools: Record<number, ProviderShared.ToolAccumulator>
157148
readonly toolCalls: ReadonlyArray<ParsedToolCall>
158149
readonly usage?: Usage
159150
readonly finishReason?: FinishReason
160151
}
161152

162-
const decodeTarget = Schema.decodeUnknownEffect(OpenAIChatDraft.pipe(Schema.decodeTo(OpenAIChatTarget)))
163-
164153
const invalid = ProviderShared.invalidRequest
165154

166155
const baseUrl = (request: LLMRequest) => ProviderShared.trimBaseUrl(request.model.baseURL ?? "https://api.openai.com/v1")
@@ -278,12 +267,12 @@ const mapUsage = (usage: OpenAIChatChunk["usage"]): Usage | undefined => {
278267
outputTokens: usage.completion_tokens,
279268
reasoningTokens: usage.completion_tokens_details?.reasoning_tokens,
280269
cacheReadInputTokens: usage.prompt_tokens_details?.cached_tokens,
281-
totalTokens: usage.total_tokens,
270+
totalTokens: ProviderShared.totalTokens(usage.prompt_tokens, usage.completion_tokens, usage.total_tokens),
282271
native: usage,
283272
})
284273
}
285274

286-
const pushToolDelta = (tools: Record<number, ToolAccumulator>, delta: OpenAIChatToolCallDelta) =>
275+
const pushToolDelta = (tools: Record<number, ProviderShared.ToolAccumulator>, delta: OpenAIChatToolCallDelta) =>
287276
Effect.gen(function* () {
288277
const current = tools[delta.index]
289278
const id = delta.id ?? current?.id
@@ -298,7 +287,7 @@ const pushToolDelta = (tools: Record<number, ToolAccumulator>, delta: OpenAIChat
298287
}
299288
})
300289

301-
const finalizeToolCalls = (tools: Record<number, ToolAccumulator>) =>
290+
const finalizeToolCalls = (tools: Record<number, ProviderShared.ToolAccumulator>) =>
302291
Effect.forEach(Object.values(tools), (tool) =>
303292
Effect.gen(function* () {
304293
const input = yield* ProviderShared.parseToolInput(ADAPTER, tool.name, tool.input)

‎packages/llm/src/provider/openai-responses.ts‎

Lines changed: 12 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -125,26 +125,16 @@ const OpenAIResponsesChunk = Schema.Struct({
125125
})
126126
type OpenAIResponsesChunk = Schema.Schema.Type<typeof OpenAIResponsesChunk>
127127

128-
const OpenAIResponsesChunkJson = Schema.fromJsonString(OpenAIResponsesChunk)
129-
const OpenAIResponsesTargetJson = Schema.fromJsonString(OpenAIResponsesTarget)
130-
const decodeChunkSync = Schema.decodeUnknownSync(OpenAIResponsesChunkJson)
131-
132-
const decodeChunk = (data: string) =>
133-
Effect.try({
134-
try: () => decodeChunkSync(data),
135-
catch: () => ProviderShared.chunkError(ADAPTER, "Invalid OpenAI Responses stream chunk", data),
136-
})
137-
const encodeTarget = Schema.encodeSync(OpenAIResponsesTargetJson)
138-
const decodeTarget = Schema.decodeUnknownEffect(OpenAIResponsesDraft.pipe(Schema.decodeTo(OpenAIResponsesTarget)))
139-
140-
interface ToolAccumulator {
141-
readonly id: string
142-
readonly name: string
143-
readonly input: string
144-
}
128+
const { encodeTarget, decodeTarget, decodeChunk } = ProviderShared.codecs({
129+
adapter: ADAPTER,
130+
draft: OpenAIResponsesDraft,
131+
target: OpenAIResponsesTarget,
132+
chunk: OpenAIResponsesChunk,
133+
chunkErrorMessage: "Invalid OpenAI Responses stream chunk",
134+
})
145135

146136
interface ParserState {
147-
readonly tools: Record<string, ToolAccumulator>
137+
readonly tools: Record<string, ProviderShared.ToolAccumulator>
148138
}
149139

150140
const invalid = ProviderShared.invalidRequest
@@ -246,7 +236,7 @@ const mapUsage = (usage: OpenAIResponsesUsage | undefined) => {
246236
outputTokens: usage.output_tokens,
247237
reasoningTokens: usage.output_tokens_details?.reasoning_tokens,
248238
cacheReadInputTokens: usage.input_tokens_details?.cached_tokens,
249-
totalTokens: usage.total_tokens ?? (usage.input_tokens ?? 0) + (usage.output_tokens ?? 0),
239+
totalTokens: ProviderShared.totalTokens(usage.input_tokens, usage.output_tokens, usage.total_tokens),
250240
native: usage,
251241
})
252242
}
@@ -258,7 +248,7 @@ const mapFinishReason = (chunk: OpenAIResponsesChunk): FinishReason => {
258248
return "unknown"
259249
}
260250

261-
const pushToolDelta = (tools: Record<string, ToolAccumulator>, itemId: string, delta: string) =>
251+
const pushToolDelta = (tools: Record<string, ProviderShared.ToolAccumulator>, itemId: string, delta: string) =>
262252
Effect.gen(function* () {
263253
const current = tools[itemId]
264254
if (!current) {
@@ -267,15 +257,15 @@ const pushToolDelta = (tools: Record<string, ToolAccumulator>, itemId: string, d
267257
return { ...current, input: `${current.input}${delta}` }
268258
})
269259

270-
const finishToolCall = (tools: Record<string, ToolAccumulator>, item: NonNullable<OpenAIResponsesChunk["item"]>) =>
260+
const finishToolCall = (tools: Record<string, ProviderShared.ToolAccumulator>, item: NonNullable<OpenAIResponsesChunk["item"]>) =>
271261
Effect.gen(function* () {
272262
if (item.type !== "function_call" || !item.id || !item.call_id || !item.name) return [] as ReadonlyArray<LLMEvent>
273263
const raw = item.arguments ?? tools[item.id]?.input ?? ""
274264
const input = yield* ProviderShared.parseToolInput(ADAPTER, item.name, raw)
275265
return [{ type: "tool-call" as const, id: item.call_id, name: item.name, input }]
276266
})
277267

278-
const withoutTool = (tools: Record<string, ToolAccumulator>, id: string | undefined) =>
268+
const withoutTool = (tools: Record<string, ProviderShared.ToolAccumulator>, id: string | undefined) =>
279269
id === undefined ? tools : Object.fromEntries(Object.entries(tools).filter(([key]) => key !== id))
280270

281271
// Hosted tool items (provider-executed) ship their typed input + status + result

‎packages/llm/src/provider/shared.ts‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,72 @@ export const Json = Schema.fromJsonString(Schema.Unknown)
88
export const decodeJson = Schema.decodeUnknownSync(Json)
99
export const encodeJson = Schema.encodeSync(Json)
1010

11+
/**
12+
* Streaming tool-call accumulator. Adapters that build a tool call across
13+
* multiple `tool-input-delta` chunks store the partial JSON input string here
14+
* and finalize it with `parseToolInput` once the call completes. Anthropic
15+
* extends this with a `providerExecuted` flag for hosted (server-side) tools;
16+
* it should be the only adapter to do so.
17+
*/
18+
export interface ToolAccumulator {
19+
readonly id: string
20+
readonly name: string
21+
readonly input: string
22+
}
23+
24+
/**
25+
* Codec bundle for a streaming JSON adapter:
26+
*
27+
* - `encodeTarget(target)` produces the JSON string body for `jsonPost`.
28+
* - `decodeTarget(draft)` runs the Schema-driven `Draft → Target` decode
29+
* inside an Effect, mapping parse errors to `InvalidRequestError` via
30+
* `validateWith` so the result drops directly into `Adapter.define`'s
31+
* `validate` field.
32+
* - `decodeChunk(input)` decodes one streaming JSON chunk against the chunk
33+
* schema. The default expects a `string` (the SSE data field); pass a
34+
* custom decoder shape via `decodeChunkInput` for adapters whose framing
35+
* already produces a parsed object (e.g. Bedrock's event-stream payloads).
36+
*
37+
* Adapters that need a totally different decode shape should still hand-roll
38+
* those pieces — the helper covers the common SSE-JSON case used by 4 of 6
39+
* adapters today.
40+
*/
41+
export const codecs = <Draft, Target, Chunk>(input: {
42+
readonly adapter: string
43+
readonly draft: Schema.Codec<Draft, unknown>
44+
readonly target: Schema.Codec<Target, unknown>
45+
readonly chunk: Schema.Codec<Chunk, unknown>
46+
readonly chunkErrorMessage: string
47+
}) => {
48+
const encodeTarget = Schema.encodeSync(Schema.fromJsonString(input.target))
49+
const decodeTarget = validateWith(
50+
Schema.decodeUnknownEffect(input.draft.pipe(Schema.decodeTo(input.target))),
51+
)
52+
const decodeChunkSync = Schema.decodeUnknownSync(Schema.fromJsonString(input.chunk))
53+
const decodeChunk = (data: string) =>
54+
Effect.try({
55+
try: () => decodeChunkSync(data),
56+
catch: () => chunkError(input.adapter, input.chunkErrorMessage, data),
57+
})
58+
return { encodeTarget, decodeTarget, decodeChunk }
59+
}
60+
61+
/**
62+
* `Usage.totalTokens` policy shared by every adapter. Honors a provider-
63+
* supplied total; otherwise falls back to `inputTokens + outputTokens` only
64+
* when at least one is defined. Returns `undefined` when neither input nor
65+
* output is known so adapters don't publish a misleading `0`.
66+
*/
67+
export const totalTokens = (
68+
inputTokens: number | undefined,
69+
outputTokens: number | undefined,
70+
total: number | undefined,
71+
) => {
72+
if (total !== undefined) return total
73+
if (inputTokens === undefined && outputTokens === undefined) return undefined
74+
return (inputTokens ?? 0) + (outputTokens ?? 0)
75+
}
76+
1177
export const chunkError = (adapter: string, message: string, raw?: string) =>
1278
new ProviderChunkError({ adapter, message, raw })
1379

0 commit comments

Comments
 (0)