Skip to content

Commit c5556a5

Browse files
core: refuse apply_diff on bytes the decoder cannot read instead of saving U+FFFD back
fs.readFile with a "utf-8" encoding replaces every byte sequence it cannot decode with U+FFFD. The tool then applies the diff to that decoded string and saves the result back through saveDirectly or saveChanges, so editing a file whose bytes are not valid UTF-8 used to rewrite bytes the model never asked to change. The read now takes the bytes and decodes them with `new TextDecoder("utf-8", { fatal: true })`, which refuses: the tool names the file and the first undecodable byte by offset on the error path it already uses for a missing file (say("error"), recordToolError, consecutiveMistakeCount, didToolFailInCurrentTurn, pushToolResult) and returns before the diff is computed, approved, or written. The decision is fail-closed, consistent with the guarded publish: a file that used to be edited lossily is now not edited at all, and the message names converting it to UTF-8 as the remediation. The three fs doubles in applyDiffTool.failureReporting.spec.ts returned a string from a call that takes no encoding, which is the lie the lossy decode was riding on; they now return Buffer and the mock is typed against the byte-returning signature it stands in for. Four rows cover the new branch: the refusal names the file and the byte, nothing is approved or written when it fires, valid multi-byte UTF-8 still edits, and repeated invalid bytes report the first offset and not a later one. They are Buffer and string logic only - no inode recycling, no cmd.exe, no junction - so nothing in them depends on how this box behaves; the ubuntu lane is what instruments this file's lane. Verification, pinned node v22.23.4 for every run below except the project pre-push hook, which ran pnpm's bundled node v25.9.0 on cached turbo tasks (the weaker of the two statements): the failureReporting suite plus its two neighbours (applyPatchTool.execute, editFileTool) pass 79/79 with stdio on file descriptors; eslint --prune-suppressions --max-warnings=0 exits 0 on both touched files and its rewrite of src/eslint-suppressions.json was reverted, so this commit carries no suppression churn; tsc --noEmit -p tsconfig.json reports one error, activate/registerCommands.ts:88, on the patched tree and the identical error on pristine 303af45, so the type delta is zero. prettier --write ran before measuring and before committing, and the number below is the committed formatted tree's: 346 of 500 counted executable lines at base 73931fe with base...head and --unified=0, against 333 at 303af45 - +13 for this change - measured with v22.23.4. The gate number is quoted here in the same sentence as the base it was taken against, because a line count without its base is not a count.
1 parent 303af45 commit c5556a5

2 files changed

Lines changed: 90 additions & 4 deletions

File tree

‎src/core/tools/ApplyDiffTool.ts‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,29 @@ export class ApplyDiffTool extends BaseTool<"apply_diff"> {
7373
return
7474
}
7575

76-
const originalContent: string = await fs.readFile(absolutePath, "utf-8")
76+
// Bytes, not a decoded string: fs.readFile with a "utf-8" encoding replaces every byte sequence
77+
// that is not valid UTF-8 with U+FFFD, and every path below republishes the decoded string -
78+
// saveDirectly, saveChanges, and the baseline the diff view and the approval payload are built
79+
// from - so a file with undecodable bytes used to be saved back with the replacements in it,
80+
// silently rewriting bytes the model never asked to touch. Decoding fatally refuses instead,
81+
// which is the fail-closed decision the guarded publish already makes about a target it cannot
82+
// vouch for. The file is reported by name and the first undecodable byte is reported by offset,
83+
// because the refusal is only actionable if the model can find the byte.
84+
const originalBytes = await fs.readFile(absolutePath)
85+
let originalContent: string
86+
try {
87+
originalContent = new TextDecoder("utf-8", { fatal: true }).decode(originalBytes)
88+
} catch {
89+
const lossy = new TextDecoder("utf-8").decode(originalBytes)
90+
const undecodableByte = new TextEncoder().encode(lossy.slice(0, lossy.indexOf("\uFFFD"))).length
91+
task.consecutiveMistakeCount++
92+
task.recordToolError("apply_diff")
93+
const formattedError = `File ${absolutePath} is not valid UTF-8 (first undecodable byte at offset ${undecodableByte}).\n\n<error_details>\napply_diff decodes the file, applies the diff to the decoded text, and saves the result back. Bytes that cannot be decoded would have been saved back as U+FFFD alongside every byte the diff did not touch, so the edit was refused and nothing was written. Re-read the file and convert it to UTF-8 before applying the diff again.\n</error_details>`
94+
await task.say("error", formattedError)
95+
task.didToolFailInCurrentTurn = true
96+
pushToolResult(formattedError)
97+
return
98+
}
7799

78100
// Apply the diff to the original content
79101
const diffResult = (await task.diffStrategy?.applyDiff(

‎src/core/tools/__tests__/applyDiffTool.failureReporting.spec.ts‎

Lines changed: 67 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,15 @@ vi.mock("@roo-code/telemetry", () => ({
1414

1515
vi.mock("fs/promises", () => ({
1616
default: {
17-
readFile: vi.fn().mockResolvedValue(""),
17+
readFile: vi.fn().mockResolvedValue(Buffer.alloc(0)),
1818
},
1919
}))
2020

21+
// ApplyDiffTool reads the file with no encoding, so the call the double stands in for is the
22+
// byte-returning one: fs.readFile returns Buffer unless an encoding is passed. vi.mocked resolves
23+
// the encoding overload, so the mock is typed against the signature the tool actually calls.
24+
const mockedReadFile = vi.mocked(fs.readFile as (path: string) => Promise<Buffer>)
25+
2126
vi.mock("../../../utils/fs", () => ({
2227
fileExistsAtPath: vi.fn().mockResolvedValue(true),
2328
}))
@@ -59,7 +64,7 @@ describe("ApplyDiffTool failure reporting", () => {
5964
vi.clearAllMocks()
6065
pushedResult = undefined
6166
vi.mocked(fileExistsAtPath).mockResolvedValue(true)
62-
vi.mocked(fs.readFile).mockResolvedValue("line one\nline two\nline three\n")
67+
mockedReadFile.mockResolvedValue(Buffer.from("line one\nline two\nline three\n", "utf8"))
6368

6469
mockAskApproval = vi.fn().mockResolvedValue(true)
6570
mockHandleError = vi.fn().mockResolvedValue(undefined)
@@ -146,6 +151,65 @@ describe("ApplyDiffTool failure reporting", () => {
146151
return failures
147152
}
148153

154+
describe("bytes the decoder cannot read", () => {
155+
async function runOnBytes(bytes: Buffer): Promise<string> {
156+
mockedReadFile.mockResolvedValue(bytes)
157+
const applyDiff = vi.fn().mockResolvedValue({ success: true, content: "unused" })
158+
mockTask.diffStrategy = { getName: () => "test", applyDiff }
159+
await applyDiffTool.execute({ path: testFilePath, diff: buildDiffContent(1) }, mockTask, {
160+
askApproval: mockAskApproval,
161+
handleError: mockHandleError,
162+
toolCallId: "tool-call-1",
163+
pushToolResult: vi.fn((result: ToolResponse) => {
164+
pushedResult = result
165+
}),
166+
})
167+
return String(pushedResult ?? "")
168+
}
169+
170+
// A byte the UTF-8 decoder rejects sits at index 3; everything around it is plain ASCII, so the
171+
// offset the tool reports has to be that index and nothing else.
172+
const UNDECODABLE = Buffer.from([0x68, 0x69, 0x20, 0xff, 0x61, 0x0a])
173+
174+
it("refuses the edit and names the file and the first undecodable byte", async () => {
175+
const output = await runOnBytes(UNDECODABLE)
176+
177+
expect(output).toContain("is not valid UTF-8")
178+
expect(output).toContain("first undecodable byte at offset 3")
179+
expect(mockTask.say).toHaveBeenCalledWith("error", output)
180+
expect(mockTask.recordToolError).toHaveBeenCalledWith("apply_diff")
181+
expect(mockTask.consecutiveMistakeCount).toBe(1)
182+
expect(mockTask.didToolFailInCurrentTurn).toBe(true)
183+
})
184+
185+
it("refuses before the diff is computed, approved, or written", async () => {
186+
await runOnBytes(UNDECODABLE)
187+
188+
expect(mockedReadFile).toHaveBeenCalledWith(path.resolve("/workspace", testFilePath))
189+
expect(mockAskApproval).not.toHaveBeenCalled()
190+
expect(mockTask.diffViewProvider.open).not.toHaveBeenCalled()
191+
expect(mockTask.diffViewProvider.saveChanges).not.toHaveBeenCalled()
192+
expect(mockTask.diffViewProvider.saveDirectly).not.toHaveBeenCalled()
193+
expect(mockTask.didEditFile).toBe(false)
194+
})
195+
196+
it("still edits a file whose bytes are valid multi-byte UTF-8", async () => {
197+
const output = await runOnBytes(Buffer.from("héllo 中文 ✅\n", "utf8"))
198+
199+
expect(output).not.toContain("is not valid UTF-8")
200+
expect(output).toContain("Changes successfully applied.")
201+
expect(mockTask.consecutiveMistakeCount).toBe(0)
202+
expect(mockTask.diffViewProvider.saveChanges).toHaveBeenCalled()
203+
})
204+
205+
it("reports the first undecodable byte when invalid bytes repeat", async () => {
206+
const output = await runOnBytes(Buffer.from([0x41, 0xc3, 0x28, 0x42, 0xfe, 0x43, 0xff]))
207+
208+
expect(output).toContain("first undecodable byte at offset 1")
209+
expect(output).not.toContain("offset 4")
210+
})
211+
})
212+
149213
describe("failure accumulation and rendering", () => {
150214
it("reports every failed block, not just the last one", async () => {
151215
const output = await runApplyDiff({ success: false, failParts: makeFailureSet(2) }, 2)
@@ -326,7 +390,7 @@ describe("ApplyDiffTool failure reporting", () => {
326390
const applyingBlock = "<<<<<<< SEARCH\nconst header = 1\n=======\nconst header = 3\n>>>>>>> REPLACE"
327391

328392
async function runRealStrategy(diff: string): Promise<string> {
329-
vi.mocked(fs.readFile).mockResolvedValue(fileText)
393+
mockedReadFile.mockResolvedValue(Buffer.from(fileText, "utf8"))
330394
const strategy = new MultiSearchReplaceDiffStrategy()
331395
const realResult = await strategy.applyDiff(fileText, diff)
332396
// Synthetic details exercise the tool's own guard against forwarding strategy detail objects.

0 commit comments

Comments
 (0)