Skip to content

Commit 2157677

Browse files
zoomote[bot]roomoteedelauna
authored
[Fix] DeepSeek/OpenAI requests fail permanently when task history contains a lone UTF-16 surrogate (Zoo-Code-Org#1831)
* fix(api): sanitize lone UTF-16 surrogates in OpenAI-chat request bodies DeepSeek rejects the entire request body when any string contains a lone UTF-16 surrogate (e.g. after history text was sliced through an astral-plane character), permanently breaking the affected task. Apply the same sanitization pattern PR Zoo-Code-Org#1605 added for the VS Code LM provider to the shared OpenAI-chat transforms, covering the DeepSeek and OpenAI providers (and other providers on the same shared path): - Move sanitizeSurrogates/sanitizeIdentifierSurrogates/ sanitizeSurrogatesDeep into a shared transform module; vscode-lm-format re-exports them. - convertToOpenAiMessages/convertToR1Format: sanitize message text, tool_result content, tool_use ids (injective, composed with normalizeToolCallId so call/result pairing survives), tool names, nested tool argument strings, and reasoning_content. - OpenAiHandler: sanitize the system/developer prompt. - BaseProvider.convertToolsForOpenAI: sanitize tool definition name/description/parameters. completePrompt is intentionally out of scope, matching Zoo-Code-Org#1605. Fixes Zoo-Code-Org#461 * test(api): make lone-surrogate body assertions effective and close coverage gaps JSON.stringify escapes lone surrogates as \udXXX text, so the copied regex-over-serialized-body assertions could never fail. Replace them with a shared expectNoLoneSurrogates helper that walks raw values recursively and fails on any unpaired UTF-16 surrogate (ES2022-safe, no isWellFormed). Also close the coverage gaps from review: - OpenAiHandler: cache_control system block with prompt caching, non-streaming system prompt, and both o3 developer-message paths, with exact sanitized text. - convertToR1Format: assistant string sanitization, assistant string merge, and user string merge into the previous user message. - convertToOpenAiMessages: text merged into the last tool message after tool_result blocks, and reasoning_content on the array-content assistant path. - convertToolsForOpenAI: mcp__srv__tool name with a lone surrogate nested in parameters, still sanitized while strict stays false. - normalizeToolCallId composition tests now use a lowercase normalizer, so swapping the sanitize/normalize order makes them fail. Ref Zoo-Code-Org#461 * test(api): cover surrogate sanitization of reasoning block extraction --------- Co-authored-by: Roomote <[email protected]> Co-authored-by: Elliott de Launay <[email protected]>
1 parent 7214352 commit 2157677

14 files changed

Lines changed: 1024 additions & 103 deletions

‎src/api/providers/__tests__/base-provider.spec.ts‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import type { ModelInfo } from "@roo-code/types"
44

55
import { BaseProvider } from "../base-provider"
66
import type { ApiStream } from "../../transform/stream"
7+
import { expectNoLoneSurrogates } from "../../../test-utils/surrogates"
78

89
// Create a concrete implementation for testing
910
class TestProvider extends BaseProvider {
@@ -267,6 +268,84 @@ describe("BaseProvider", () => {
267268
expect(result?.[0].function.parameters.additionalProperties).toBeUndefined()
268269
})
269270

271+
it("should sanitize lone UTF-16 surrogates in name, description, and parameters (#461)", () => {
272+
const tools = [
273+
{
274+
type: "function",
275+
function: {
276+
name: "read_file",
277+
description: "bad\uD800end",
278+
parameters: {
279+
type: "object",
280+
properties: {
281+
path: { type: "string", description: "bad\uDC00end" },
282+
},
283+
},
284+
},
285+
},
286+
]
287+
288+
const result = provider.testConvertToolsForOpenAI(tools)
289+
290+
expect(result?.[0].function.description).toBe("bad\uFFFDend")
291+
expect(result?.[0].function.parameters.properties.path.description).toBe("bad\uFFFDend")
292+
// Inspect the raw values: JSON.stringify escapes lone surrogates as \udXXX text,
293+
// so a regex over the serialized body can never fail. See expectNoLoneSurrogates.
294+
expectNoLoneSurrogates(result)
295+
})
296+
297+
it("should sanitize lone UTF-16 surrogates in nested MCP tool parameters (#461)", () => {
298+
const lone = "bad\uD800end"
299+
const sanitized = "bad\uFFFDend"
300+
const tools = [
301+
{
302+
type: "function",
303+
function: {
304+
name: "mcp__srv__tool",
305+
description: "Run an MCP tool",
306+
parameters: {
307+
type: "object",
308+
properties: {
309+
path: { type: "string" },
310+
nested: {
311+
type: "object",
312+
properties: {
313+
list: { type: "string", description: lone },
314+
},
315+
},
316+
},
317+
},
318+
},
319+
},
320+
]
321+
322+
const result = provider.testConvertToolsForOpenAI(tools)
323+
324+
// MCP tools keep their original schema (no strict-mode conversion) ...
325+
expect(result?.[0].function.strict).toBe(false)
326+
expect(result?.[0].function.parameters.additionalProperties).toBeUndefined()
327+
// ... but strings nested in the parameters are still sanitized.
328+
expect(result?.[0].function.parameters.properties.nested.properties.list.description).toBe(sanitized)
329+
expectNoLoneSurrogates(result)
330+
})
331+
332+
it("should sanitize lone UTF-16 surrogates in tool names (#461)", () => {
333+
const tools = [
334+
{
335+
type: "function",
336+
function: {
337+
name: "read\uD800file",
338+
description: "Read a file",
339+
parameters: { type: "object", properties: {} },
340+
},
341+
},
342+
]
343+
344+
const result = provider.testConvertToolsForOpenAI(tools)
345+
346+
expect(result?.[0].function.name).toBe("read\uFFFDfile")
347+
})
348+
270349
it("should preserve non-function tools unchanged", () => {
271350
const tools = [
272351
{

‎src/api/providers/__tests__/deepseek.spec.ts‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Mocks must come first, before imports
22
import { asyncStreamFrom, collectStream } from "../../../test-utils/stream"
33
import { clearAllMocks } from "../../../test-utils/reset"
4+
import { expectNoLoneSurrogates } from "../../../test-utils/surrogates"
45

56
const mockCreate = vi.fn()
67
vi.mock("openai", () => {
@@ -824,3 +825,69 @@ describe("DeepSeekHandler", () => {
824825
})
825826
})
826827
})
828+
829+
describe("DeepSeekHandler lone surrogate sanitization (#461)", () => {
830+
beforeEach(() => {
831+
mockCreate.mockClear()
832+
})
833+
834+
const surrogateOptions: ApiHandlerOptions = {
835+
deepSeekApiKey: "test-api-key",
836+
apiModelId: "deepseek-v4-flash",
837+
deepSeekBaseUrl: "https://api.deepseek.com",
838+
}
839+
840+
it("sends a request body free of lone surrogates, keeping tool call/result pairing intact", async () => {
841+
const lone = "bad\uD800end"
842+
const sanitized = "bad\uFFFDend"
843+
const handler = new DeepSeekHandler(surrogateOptions)
844+
const messages: Anthropic.Messages.MessageParam[] = [
845+
{ role: "user", content: lone },
846+
{
847+
role: "assistant",
848+
content: [{ type: "tool_use", id: "call-\uD800", name: "read_file", input: { path: lone } }],
849+
},
850+
{ role: "user", content: [{ type: "tool_result", tool_use_id: "call-\uD800", content: lone }] },
851+
]
852+
853+
await collectStream(
854+
handler.createMessage("system \uD800 prompt", messages, {
855+
taskId: "task-1",
856+
tools: [
857+
{
858+
type: "function",
859+
function: {
860+
name: "read_file",
861+
description: lone,
862+
parameters: {
863+
type: "object",
864+
properties: { path: { type: "string", description: lone } },
865+
},
866+
},
867+
},
868+
],
869+
}),
870+
)
871+
872+
expect(mockCreate).toHaveBeenCalledOnce()
873+
const request = mockCreate.mock.calls[0][0]
874+
875+
// The whole body (messages + tools) must be free of lone surrogates. Inspect the raw
876+
// values: JSON.stringify escapes lone surrogates as \udXXX text, so a regex over the
877+
// serialized body can never fail. See expectNoLoneSurrogates.
878+
expectNoLoneSurrogates(request)
879+
880+
// The tool call and its result stay paired after injective id sanitization.
881+
const assistantMessage = request.messages.find(
882+
(message: { role: string }) => message.role === "assistant",
883+
) as OpenAI.Chat.ChatCompletionAssistantMessageParam
884+
const toolMessage = request.messages.find(
885+
(message: { role: string }) => message.role === "tool",
886+
) as OpenAI.Chat.ChatCompletionToolMessageParam
887+
const toolCalls = assistantMessage.tool_calls as OpenAI.Chat.ChatCompletionMessageFunctionToolCall[]
888+
expect(toolMessage.tool_call_id).toBe(toolCalls[0].id)
889+
expect(toolMessage.content).toBe(sanitized)
890+
expect(JSON.parse(toolCalls[0].function.arguments)).toEqual({ path: sanitized })
891+
expect(request.tools[0].function.description).toBe(sanitized)
892+
})
893+
})

‎src/api/providers/__tests__/openai.spec.ts‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import {
1313
import { Package } from "../../../shared/package"
1414
import { makeApiHandlerOptions } from "../../../test-utils/api"
1515
import { asyncStreamFrom, collectStream } from "../../../test-utils/stream"
16+
import { expectNoLoneSurrogates } from "../../../test-utils/surrogates"
1617
import axios from "axios"
1718

1819
vitest.mock("../utils/timeout-config", () => ({
@@ -1877,3 +1878,163 @@ describe("getOpenAiModels", () => {
18771878
expect(result).toEqual(["gpt-4", "gpt-3.5-turbo"])
18781879
})
18791880
})
1881+
1882+
describe("OpenAiHandler lone surrogate sanitization (#461)", () => {
1883+
beforeEach(() => {
1884+
mockCreate.mockClear()
1885+
})
1886+
1887+
it("sends a request body free of lone surrogates, keeping tool call/result pairing intact", async () => {
1888+
const lone = "bad\uD800end"
1889+
const sanitized = "bad\uFFFDend"
1890+
const handler = new OpenAiHandler(
1891+
makeApiHandlerOptions({
1892+
openAiApiKey: "test-api-key",
1893+
openAiModelId: "gpt-4",
1894+
openAiBaseUrl: "https://api.openai.com/v1",
1895+
}),
1896+
)
1897+
const messages: Anthropic.Messages.MessageParam[] = [
1898+
{ role: "user", content: lone },
1899+
{
1900+
role: "assistant",
1901+
content: [{ type: "tool_use", id: "call-\uD800", name: "read_file", input: { path: lone } }],
1902+
},
1903+
{ role: "user", content: [{ type: "tool_result", tool_use_id: "call-\uD800", content: lone }] },
1904+
]
1905+
1906+
await collectStream(
1907+
handler.createMessage("system \uD800 prompt", messages, {
1908+
taskId: "task-1",
1909+
tools: [
1910+
{
1911+
type: "function",
1912+
function: {
1913+
name: "read_file",
1914+
description: lone,
1915+
parameters: {
1916+
type: "object",
1917+
properties: { path: { type: "string", description: lone } },
1918+
},
1919+
},
1920+
},
1921+
],
1922+
}),
1923+
)
1924+
1925+
expect(mockCreate).toHaveBeenCalledOnce()
1926+
const request = mockCreate.mock.calls[0][0]
1927+
1928+
// The whole body (system prompt, messages, tools) must be free of lone surrogates.
1929+
// Inspect the raw values: JSON.stringify escapes lone surrogates as \udXXX text, so
1930+
// a regex over the serialized body can never fail. See expectNoLoneSurrogates.
1931+
expectNoLoneSurrogates(request)
1932+
1933+
expect(request.messages[0]).toEqual({ role: "system", content: "system \uFFFD prompt" })
1934+
1935+
// The tool call and its result stay paired after injective id sanitization.
1936+
const assistantMessage = request.messages.find(
1937+
(message: { role: string }) => message.role === "assistant",
1938+
) as OpenAI.Chat.ChatCompletionAssistantMessageParam
1939+
const toolMessage = request.messages.find(
1940+
(message: { role: string }) => message.role === "tool",
1941+
) as OpenAI.Chat.ChatCompletionToolMessageParam
1942+
const toolCalls = assistantMessage.tool_calls as OpenAI.Chat.ChatCompletionMessageFunctionToolCall[]
1943+
expect(toolMessage.tool_call_id).toBe(toolCalls[0].id)
1944+
expect(toolMessage.content).toBe(sanitized)
1945+
expect(JSON.parse(toolCalls[0].function.arguments)).toEqual({ path: sanitized })
1946+
expect(request.tools[0].function.description).toBe(sanitized)
1947+
})
1948+
1949+
it("sanitizes the system prompt inside the cache_control block when streaming with prompt caching", async () => {
1950+
const handler = new OpenAiHandler(
1951+
makeApiHandlerOptions({
1952+
openAiApiKey: "test-api-key",
1953+
openAiModelId: "gpt-4",
1954+
openAiBaseUrl: "https://api.openai.com/v1",
1955+
openAiCustomModelInfo: { ...openAiModelInfoSaneDefaults, supportsPromptCache: true },
1956+
}),
1957+
)
1958+
1959+
await collectStream(
1960+
handler.createMessage("system \uD800 prompt", [{ role: "user", content: "hi" }], { taskId: "task-1" }),
1961+
)
1962+
1963+
expect(mockCreate).toHaveBeenCalledOnce()
1964+
const request = mockCreate.mock.calls[0][0]
1965+
expect(request.messages[0]).toEqual({
1966+
role: "system",
1967+
content: [{ type: "text", text: "system \uFFFD prompt", cache_control: { type: "ephemeral" } }],
1968+
})
1969+
expect(request.stream).toBe(true)
1970+
expectNoLoneSurrogates(request)
1971+
})
1972+
1973+
it("sanitizes the system prompt string for non-streaming requests", async () => {
1974+
const handler = new OpenAiHandler(
1975+
makeApiHandlerOptions({
1976+
openAiApiKey: "test-api-key",
1977+
openAiModelId: "gpt-4",
1978+
openAiBaseUrl: "https://api.openai.com/v1",
1979+
openAiStreamingEnabled: false,
1980+
}),
1981+
)
1982+
1983+
await collectStream(
1984+
handler.createMessage("system \uD800 prompt", [{ role: "user", content: "hi" }], { taskId: "task-1" }),
1985+
)
1986+
1987+
expect(mockCreate).toHaveBeenCalledOnce()
1988+
const request = mockCreate.mock.calls[0][0]
1989+
expect(request.messages[0]).toEqual({ role: "system", content: "system \uFFFD prompt" })
1990+
expect(request.stream).toBeUndefined()
1991+
expectNoLoneSurrogates(request)
1992+
})
1993+
1994+
it("sanitizes the o3 developer message when streaming", async () => {
1995+
const handler = new OpenAiHandler(
1996+
makeApiHandlerOptions({
1997+
openAiApiKey: "test-api-key",
1998+
openAiModelId: "o3-mini",
1999+
openAiBaseUrl: "https://api.openai.com/v1",
2000+
}),
2001+
)
2002+
2003+
await collectStream(
2004+
handler.createMessage("system \uD800 prompt", [{ role: "user", content: "hi" }], { taskId: "task-1" }),
2005+
)
2006+
2007+
expect(mockCreate).toHaveBeenCalledOnce()
2008+
const request = mockCreate.mock.calls[0][0]
2009+
expect(request.messages[0]).toEqual({
2010+
role: "developer",
2011+
content: "Formatting re-enabled\nsystem \uFFFD prompt",
2012+
})
2013+
expect(request.stream).toBe(true)
2014+
expectNoLoneSurrogates(request)
2015+
})
2016+
2017+
it("sanitizes the o3 developer message for non-streaming requests", async () => {
2018+
const handler = new OpenAiHandler(
2019+
makeApiHandlerOptions({
2020+
openAiApiKey: "test-api-key",
2021+
openAiModelId: "o3-mini",
2022+
openAiBaseUrl: "https://api.openai.com/v1",
2023+
openAiStreamingEnabled: false,
2024+
}),
2025+
)
2026+
2027+
await collectStream(
2028+
handler.createMessage("system \uD800 prompt", [{ role: "user", content: "hi" }], { taskId: "task-1" }),
2029+
)
2030+
2031+
expect(mockCreate).toHaveBeenCalledOnce()
2032+
const request = mockCreate.mock.calls[0][0]
2033+
expect(request.messages[0]).toEqual({
2034+
role: "developer",
2035+
content: "Formatting re-enabled\nsystem \uFFFD prompt",
2036+
})
2037+
expect(request.stream).toBeUndefined()
2038+
expectNoLoneSurrogates(request)
2039+
})
2040+
})

‎src/api/providers/base-provider.ts‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import type { ModelInfo } from "@roo-code/types"
44

55
import type { ApiHandler, ApiHandlerCreateMessageMetadata } from "../index"
66
import { ApiStream } from "../transform/stream"
7+
import { sanitizeSurrogates, sanitizeSurrogatesDeep } from "../transform/sanitize-surrogates"
78
import { countTokens } from "../../utils/countTokens"
89
import { isMcpTool } from "../../utils/mcp-name"
910
import { getApiRequestTimeout } from "./utils/timeout-config"
@@ -45,10 +46,14 @@ export abstract class BaseProvider implements ApiHandler {
4546
...tool,
4647
function: {
4748
...tool.function,
49+
// Sanitize lone UTF-16 surrogates: providers validating the JSON body
50+
// (e.g. DeepSeek) reject the whole request otherwise. See #461.
51+
name: sanitizeSurrogates(tool.function.name),
52+
description: sanitizeSurrogates(tool.function.description),
4853
strict: !isMcp,
49-
parameters: isMcp
50-
? tool.function.parameters
51-
: this.convertToolSchemaForOpenAI(tool.function.parameters),
54+
parameters: sanitizeSurrogatesDeep(
55+
isMcp ? tool.function.parameters : this.convertToolSchemaForOpenAI(tool.function.parameters),
56+
),
5257
},
5358
}
5459
})

0 commit comments

Comments
 (0)