Skip to content

Commit 538d755

Browse files
Yuxin-Qiaolidge-jun
authored andcommitted
fix(anthropic): complete AgentRouter streams that end before terminal frames (#658)
1 parent ab6a1ed commit 538d755

9 files changed

Lines changed: 197 additions & 3 deletions

File tree

‎docs-site/src/content/docs/ja/reference/configuration/providers.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ description: プロバイダー エントリ、認証、エンドポイント、
8282
| `thinkingBudgetModels?` | `string[]` |整数 `thinking_budget` を使用したチャット モデル。労力は予算の一部にマッピングされます。 |
8383
| `noVisionModels?` | `string[]` |ビジョン サイドカーを通じて送信されるテキストのみのモデル。マッチングでは、Ollama `:size` タグが許容されます。 |
8484
| `escapeBuiltinToolNames?` | `boolean` | Anthropic 互換ゲートウェイの組み込みツール名をエスケープし、返された呼び出しで復元します。 |
85+
| `anthropicEofTolerance?` | `boolean` | `message_stop` 前にストリームが終了しても、可視テキストまたは完全な JSON オブジェクトのツール入力が受信済みの場合に限り完了を許可します(Anthropic 互換ゲートウェイ向け)。デフォルトはオフ。 |
8586
| `googleMode?` | `"ai-studio" \| "vertex" \| "cloud-code-assist"` | Google トランスポート/認証モード。デフォルトは`ai-studio`です。 |
8687
| `project?` | `string` | Vertex または Antigravity Cloud Code Assist プロジェクト ID。 |
8788
| `location?` | `string` |頂点の位置。環境フォールバックは `GOOGLE_CLOUD_LOCATION` です。 |

‎docs-site/src/content/docs/ko/reference/configuration/providers.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ description: 공급자 항목, 인증, 엔드포인트, 모델 카탈로그, 할
8282
| `thinkingBudgetModels?` | `string[]` | 정수 `thinking_budget`를 쓰는 chat 모델입니다. effort는 예산 비율로 매핑됩니다. |
8383
| `noVisionModels?` | `string[]` | vision sidecar로 보내는 텍스트 전용 모델입니다. 일치 판정은 Ollama `:size` 태그도 허용합니다. |
8484
| `escapeBuiltinToolNames?` | `boolean` | Anthropic 호환 게이트웨이를 위해 내장 도구 이름을 이스케이프하고, 반환된 호출에서는 다시 복원합니다. |
85+
| `anthropicEofTolerance?` | `boolean` | `message_stop` 전에 스트림이 끝나도 표시 텍스트 또는 완전한 JSON 객체 툴 입력을 받은 경우에만 완료를 허용합니다(Anthropic 호환 게이트웨이용). 기본값은 꺼짐. |
8586
| `googleMode?` | `"ai-studio" \| "vertex" \| "cloud-code-assist"` | Google 전송/인증 모드입니다. 기본값은 `ai-studio`입니다. |
8687
| `project?` | `string` | Vertex 또는 Antigravity Cloud Code Assist 프로젝트 id입니다. |
8788
| `location?` | `string` | Vertex 위치입니다. 환경 변수 폴백은 `GOOGLE_CLOUD_LOCATION`입니다. |

‎docs-site/src/content/docs/reference/configuration/providers.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ differing backup and rewrites known legacy namespaced selected ids to bare ids.
9393
| `thinkingBudgetModels?` | `string[]` | Chat models using integer `thinking_budget`; effort maps to a budget fraction. |
9494
| `noVisionModels?` | `string[]` | Text-only models sent through the vision sidecar; matching tolerates an Ollama `:size` tag. |
9595
| `escapeBuiltinToolNames?` | `boolean` | Escape built-in tool names for Anthropic-compatible gateways and restore them in returned calls. |
96+
| `anthropicEofTolerance?` | `boolean` | Let an Anthropic-compatible gateway complete a stream that ends before `message_stop`, only when visible text or a complete JSON-object tool input was received. Off by default. |
9697
| `googleMode?` | `"ai-studio" \| "vertex" \| "cloud-code-assist"` | Google transport/auth mode. Default `ai-studio`. |
9798
| `project?` | `string` | Vertex or Antigravity Cloud Code Assist project id. |
9899
| `location?` | `string` | Vertex location; environment fallback is `GOOGLE_CLOUD_LOCATION`. |

‎docs-site/src/content/docs/ru/reference/configuration/providers.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ cross-route credential fallback не существует. Строки API GPT-
9898
| `thinkingBudgetModels?` | `string[]` | Chat-модели, использующие целочисленный `thinking_budget`; effort отображается в долю бюджета. |
9999
| `noVisionModels?` | `string[]` | Text-only-модели, идущие через vision sidecar; при сопоставлении tolerируется тег Ollama вида `:size`. |
100100
| `escapeBuiltinToolNames?` | `boolean` | Экранировать built-in tool name'ы для Anthropic-compatible gateway'ев и восстанавливать их в возвращаемых call'ах. |
101+
| `anthropicEofTolerance?` | `boolean` | Позволяет Anthropic-совместимому шлюзу завершить поток до `message_stop`, только если получен видимый текст или полный JSON-объект аргументов инструмента. По умолчанию выключено. |
101102
| `googleMode?` | `"ai-studio" \| "vertex" \| "cloud-code-assist"` | Режим транспорта/аутентификации Google. По умолчанию `ai-studio`. |
102103
| `project?` | `string` | Идентификатор проекта Vertex или Antigravity Cloud Code Assist. |
103104
| `location?` | `string` | Локация Vertex; fallback через окружение — `GOOGLE_CLOUD_LOCATION`. |

‎docs-site/src/content/docs/zh-cn/reference/configuration/providers.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ description: 提供者条目、身份验证、端点、模型目录、配额、
8282
| `thinkingBudgetModels?` | `string[]` | 使用整数 `thinking_budget` 的 chat 模型;effort 会映射为预算比例。 |
8383
| `noVisionModels?` | `string[]` | 经由视觉 sidecar 发送的纯文本模型;匹配时会容忍 Ollama 的 `:size` 标记。 |
8484
| `escapeBuiltinToolNames?` | `boolean` | 为 Anthropic 兼容网关转义内置工具名,并在返回的调用中恢复。 |
85+
| `anthropicEofTolerance?` | `boolean` | 允许 Anthropic 兼容网关在 `message_stop` 前结束流,仅当已收到可见文本或完整的 JSON 对象工具输入时。默认关闭。 |
8586
| `googleMode?` | `"ai-studio" \| "vertex" \| "cloud-code-assist"` | Google 传输/身份验证模式。默认 `ai-studio`。 |
8687
| `project?` | `string` | Vertex 或 Antigravity Cloud Code Assist 项目 id。 |
8788
| `location?` | `string` | Vertex 位置;环境变量回退为 `GOOGLE_CLOUD_LOCATION`。 |

‎src/adapters/anthropic.ts‎

Lines changed: 60 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -277,14 +277,50 @@ function usableToolUseId(id: unknown): string {
277277
return typeof id === "string" && id.trim() ? id : synthesizeToolUseId();
278278
}
279279

280-
function toolUseArguments(input: unknown): string {
280+
/**
281+
* Bound repair for a malformed tool-arguments string under the compatibility profile (#658):
282+
* a gateway such as AgentRouter can concatenate JSON objects (`{}{"value":42}`). Find the
283+
* last parseable JSON object by scanning suffixes from each object-open brace and prefixes
284+
* ending at each object-close brace, bounded so hostile input cannot cost unbounded time.
285+
*/
286+
function lastValidJsonObject(input: string, maxCandidates: number): string | undefined {
287+
const opens: number[] = [];
288+
const closes: number[] = [];
289+
for (let i = 0; i < input.length; i++) {
290+
if (input[i] === "{") opens.push(i);
291+
else if (input[i] === "}") closes.push(i);
292+
}
293+
let tried = 0;
294+
for (let i = opens.length - 1; i >= 0 && tried < maxCandidates; i--, tried++) {
295+
const candidate = input.slice(opens[i]);
296+
try {
297+
const parsed = JSON.parse(candidate) as unknown;
298+
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) return candidate;
299+
} catch { /* keep scanning */ }
300+
}
301+
tried = 0;
302+
for (let i = closes.length - 1; i >= 0 && tried < maxCandidates; i--, tried++) {
303+
const candidate = input.slice(0, closes[i] + 1);
304+
try {
305+
const parsed = JSON.parse(candidate) as unknown;
306+
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) return candidate;
307+
} catch { /* keep scanning */ }
308+
}
309+
return undefined;
310+
}
311+
312+
function toolUseArguments(input: unknown, lenient = false): string {
281313
if (typeof input === "string") {
282314
const trimmed = input.trim();
283315
if (!trimmed) return "{}";
284316
try {
285317
JSON.parse(trimmed);
286318
return trimmed;
287319
} catch {
320+
if (lenient) {
321+
const repaired = lastValidJsonObject(trimmed, 32);
322+
if (repaired !== undefined) return repaired;
323+
}
288324
// A tool call's arguments must be a JSON object. Re-encoding an unparseable string as a
289325
// JSON *string* is the double-encoding #765 reports: the caller then receives
290326
// `"get weather"` where an object was required and the tool call is unusable either way.
@@ -798,6 +834,7 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti
798834
let pendingUsage: Record<string, number> | undefined;
799835
let pendingStopReason: string | undefined;
800836
let emittedDone = false;
837+
let sawVisibleText = false;
801838

802839
const emitDone = function* (): Generator<AdapterEvent> {
803840
if (emittedDone) return;
@@ -853,6 +890,7 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti
853890
const delta = data.delta as Record<string, unknown> | undefined;
854891
if (!delta) break;
855892
if (delta.type === "text_delta" && typeof delta.text === "string") {
893+
sawVisibleText = true;
856894
yield { type: "text_delta", text: delta.text };
857895
} else if (delta.type === "thinking_delta" && typeof delta.thinking === "string") {
858896
yield { type: "thinking_delta", thinking: delta.thinking };
@@ -951,6 +989,26 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti
951989
usage: usageFromAnthropic(pendingUsage),
952990
...(stopReason ? { stopReason } : {}),
953991
};
992+
} else if (provider.anthropicEofTolerance === true) {
993+
// AgentRouter-style compatibility profile (#658): the upstream can close the stream
994+
// after valid content without terminal frames. Complete only when visible text was
995+
// received or an open tool call has complete JSON-object arguments; everything else
996+
// (incomplete tool JSON, no usable content, transport failure) stays a truncation
997+
// error, matching the strict default.
998+
if (currentToolCallId) {
999+
if (streamedToolArgumentsParse(currentToolCallJson)) {
1000+
budget.closeCall(currentToolCallId);
1001+
currentToolCallId = "";
1002+
yield { type: "tool_call_end" };
1003+
yield* emitDone();
1004+
} else {
1005+
yield { type: "error", message: "upstream stream ended before message_stop — possible truncation" };
1006+
}
1007+
} else if (sawVisibleText) {
1008+
yield* emitDone();
1009+
} else {
1010+
yield { type: "error", message: "upstream stream ended before message_stop — possible truncation" };
1011+
}
9541012
} else {
9551013
yield { type: "error", message: "upstream stream ended before message_stop — possible truncation" };
9561014
}
@@ -980,7 +1038,7 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti
9801038
} else if (block.type === "tool_use") {
9811039
const id = usableToolUseId(block.id);
9821040
events.push({ type: "tool_call_start", id, name: toolNames.fromWire(block.name ?? "") });
983-
events.push({ type: "tool_call_delta", arguments: toolUseArguments(block.input) });
1041+
events.push({ type: "tool_call_delta", arguments: toolUseArguments(block.input, provider.anthropicEofTolerance === true) });
9841042
events.push({ type: "tool_call_end" });
9851043
}
9861044
}

‎src/providers/free-directory.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ export interface FreeDirectoryProvider {
4545
keyOptional?: boolean;
4646
models?: string[];
4747
liveModels: boolean;
48+
/** Anthropic-compatible gateways that may close streams before terminal frames. */
49+
anthropicEofTolerance?: boolean;
4850
note?: string;
4951
googleMode?: "ai-studio" | "vertex";
5052
}
@@ -116,7 +118,7 @@ const CONNECTABLE: Record<string, ConnectableOverride> = {
116118
// `unverified` and drops the shared verification date rather than borrowing it.
117119
bytez: openAi("https://api.bytez.com/models/v2/openai/v1", "https://bytez.com", { verification: "unverified", lastVerified: undefined, documentationUrl: "https://docs.bytez.com/", discovery: "static", liveModels: false, models: ["meta-llama/Llama-3.3-70B-Instruct", "mistralai/Mistral-7B-Instruct-v0.3", "Qwen/Qwen2.5-72B-Instruct"], note: "The recurring-credit classification is retained from the requested catalog, but the current reset terms could not be independently verified." }),
118120
"nous-research": openAi("https://inference-api.nousresearch.com/v1", "https://portal.nousresearch.com", { discovery: "static", liveModels: false, models: ["Hermes-4-405B", "Hermes-4-70B"] }),
119-
agentrouter: { baseUrl: "https://agentrouter.org", dashboardUrl: "https://agentrouter.org", adapter: "anthropic", authKind: "key", supportLevel: "experimental", verification: "primary", modelsUrl: "https://agentrouter.org/v1/models", lastVerified: LAST_VERIFIED, discovery: "live", liveModels: true },
121+
agentrouter: { baseUrl: "https://agentrouter.org", dashboardUrl: "https://agentrouter.org", adapter: "anthropic", authKind: "key", supportLevel: "experimental", verification: "primary", modelsUrl: "https://agentrouter.org/v1/models", lastVerified: LAST_VERIFIED, discovery: "live", liveModels: true, anthropicEofTolerance: true },
120122
ai21: openAi("https://api.ai21.com/studio/v1", "https://studio.ai21.com/account/api-key", { supportLevel: "supported", verification: "official", documentationUrl: "https://docs.ai21.com/reference/models" }),
121123
baichuan: openAi("https://api.baichuan-ai.com/v1", "https://platform.baichuan-ai.com/console/apikey", { verification: "official" }),
122124
// Verified end-to-end 2026-07-30: /v1/models returns the OpenAI-shaped live catalog (13 models),

‎src/types.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1122,6 +1122,13 @@ export interface OcxProviderConfig {
11221122
thinkingBudgetModels?: string[];
11231123
/** Anthropic-compatible gateways that need custom tool names escaped on the wire. */
11241124
escapeBuiltinToolNames?: boolean;
1125+
/**
1126+
* Anthropic-compatible gateways (e.g. AgentRouter) that may close the stream before
1127+
* `message_stop`. With this enabled the adapter completes an otherwise-clean EOF only when
1128+
* visible text was received or an open tool call has complete JSON-object arguments; all
1129+
* other EOFs remain truncation errors. Absent = strict default behavior.
1130+
*/
1131+
anthropicEofTolerance?: boolean;
11251132
/**
11261133
* Model ids that do NOT accept image inputs. The proxy gives them "eyes" via the vision sidecar:
11271134
* attached images are described by a gpt vision model and replaced with text before the call.
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
import { describe, expect, test } from "bun:test";
2+
import { createAnthropicAdapter as createAnthropicAdapterProduction } from "../src/adapters/anthropic";
3+
import { FREE_PROVIDER_DIRECTORY } from "../src/providers/free-directory";
4+
import type { AdapterEvent, OcxProviderConfig } from "../src/types";
5+
import { withTestTranslatorBudget } from "./helpers/translator-budget";
6+
7+
/**
8+
* #658: AgentRouter's Anthropic-compatible endpoint can close the stream before
9+
* `content_block_stop`, `message_delta`, and `message_stop`. The default adapter treats
10+
* that EOF as a fatal truncation; with `anthropicEofTolerance` enabled it may complete
11+
* only when visible text was received or an open tool call has complete JSON-object
12+
* arguments. These tests pin the wire behavior; no request reaches agentrouter.org.
13+
*/
14+
15+
const createAnthropicAdapter = (...args: Parameters<typeof createAnthropicAdapterProduction>) =>
16+
withTestTranslatorBudget(createAnthropicAdapterProduction(...args));
17+
18+
function providerFor(extra: Partial<OcxProviderConfig> = {}): OcxProviderConfig {
19+
return {
20+
adapter: "anthropic",
21+
baseUrl: "https://agentrouter.org",
22+
apiKey: "test-key",
23+
authMode: "key",
24+
...extra,
25+
} as OcxProviderConfig;
26+
}
27+
28+
const strict = providerFor();
29+
const tolerant = providerFor({ anthropicEofTolerance: true });
30+
31+
const TRUNCATION = "upstream stream ended before message_stop — possible truncation";
32+
33+
function sseResponse(events: string[]): Response {
34+
return new Response(events.join("\n\n"), { headers: { "content-type": "text/event-stream" } });
35+
}
36+
37+
async function collect(provider: OcxProviderConfig, events: string[]): Promise<AdapterEvent[]> {
38+
const out: AdapterEvent[] = [];
39+
for await (const event of createAnthropicAdapter(provider).parseStream(sseResponse(events))) out.push(event);
40+
return out;
41+
}
42+
43+
const textEof = [
44+
'event: message_start\ndata: {"type":"message_start","message":{"usage":{"input_tokens":2}}}',
45+
'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}',
46+
'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"visible"}}',
47+
];
48+
49+
function toolEof(partialJson: string, id = "toolu_1"): string[] {
50+
return [
51+
'event: message_start\ndata: {"type":"message_start","message":{}}',
52+
`event: content_block_start\ndata: ${JSON.stringify({ type: "content_block_start", index: 0, content_block: { type: "tool_use", id, name: "get_weather" } })}`,
53+
`event: content_block_delta\ndata: ${JSON.stringify({ type: "content_block_delta", index: 0, delta: { type: "input_json_delta", partial_json: partialJson } })}`,
54+
];
55+
}
56+
57+
describe("AgentRouter Anthropic EOF tolerance (#658)", () => {
58+
test("text EOF completes when anthropicEofTolerance is enabled", async () => {
59+
const events = await collect(tolerant, textEof);
60+
61+
expect(events).toContainEqual({ type: "text_delta", text: "visible" });
62+
expect(events.at(-1)).toEqual({ type: "done", usage: { inputTokens: 2, outputTokens: 0 } });
63+
expect(events.some(event => event.type === "error")).toBe(false);
64+
});
65+
66+
test("the same EOF without the capability stays a truncation error", async () => {
67+
const events = await collect(strict, textEof);
68+
69+
expect(events.at(-1)).toEqual({ type: "error", message: TRUNCATION });
70+
expect(events.some(event => event.type === "done")).toBe(false);
71+
});
72+
73+
test("a complete tool call at EOF closes and completes", async () => {
74+
const events = await collect(tolerant, toolEof('{"value":42}'));
75+
76+
expect(events).toContainEqual({ type: "tool_call_start", id: "toolu_1", name: "get_weather" });
77+
expect(events).toContainEqual({ type: "tool_call_delta", arguments: '{"value":42}' });
78+
expect(events.at(-1)).toEqual({ type: "done", usage: undefined });
79+
expect(events.some(event => event.type === "error")).toBe(false);
80+
});
81+
82+
test("an incomplete tool call at EOF remains a truncation error", async () => {
83+
const events = await collect(tolerant, toolEof('{"value":'));
84+
85+
expect(events.at(-1)).toEqual({ type: "error", message: TRUNCATION });
86+
expect(events.some(event => event.type === "done" || event.type === "tool_call_end")).toBe(false);
87+
});
88+
89+
test("EOF before any usable content remains a truncation error", async () => {
90+
const events = await collect(tolerant, [
91+
'event: message_start\ndata: {"type":"message_start","message":{}}',
92+
]);
93+
94+
expect(events.at(-1)).toEqual({ type: "error", message: TRUNCATION });
95+
});
96+
97+
test("a missing tool_use id gets a stable synthesized id on the tolerant path", async () => {
98+
const events = await collect(tolerant, toolEof('{"value":42}', ""));
99+
const start = events.find(event => event.type === "tool_call_start");
100+
101+
expect(start?.type).toBe("tool_call_start");
102+
expect((start as { id: string }).id).toMatch(/^toolu_[0-9a-f]{24}$/);
103+
expect(events.at(-1)).toEqual({ type: "done", usage: undefined });
104+
});
105+
106+
test("non-stream concatenated tool input keeps the last valid object when enabled", async () => {
107+
const payload = JSON.stringify({
108+
content: [{ type: "tool_use", id: "toolu_1", name: "get_weather", input: '{}{"value":42}' }],
109+
});
110+
111+
const tolerantEvents = await createAnthropicAdapter(tolerant).parseResponse(new Response(payload));
112+
expect(tolerantEvents).toContainEqual({ type: "tool_call_delta", arguments: '{"value":42}' });
113+
114+
const strictEvents = await createAnthropicAdapter(strict).parseResponse(new Response(payload));
115+
expect(strictEvents).toContainEqual({ type: "tool_call_delta", arguments: "{}" });
116+
});
117+
118+
test("the AgentRouter directory row declares the EOF tolerance capability", () => {
119+
const row = FREE_PROVIDER_DIRECTORY.find(provider => provider.id === "agentrouter");
120+
expect(row?.anthropicEofTolerance).toBe(true);
121+
});
122+
});

0 commit comments

Comments
 (0)