Skip to content

Commit eb15d53

Browse files
tools: keep a BOM through the fatal decode and report the right bad-byte offset
Two things the fatal decode got wrong and both are mine, from c5556a5 and d5c31a1: * The BOM. `new TextDecoder("utf-8")` consumes a leading byte-order mark by default, so switching from `fs.readFile(path, "utf-8")` - which keeps it - to a decoder dropped it, and every path below republishes the decoded string: the approval baseline, `saveChanges`, `saveDirectly`. Measured on pinned v22.23.4 against a file whose bytes are EF BB BD 68 69 0D 0A 74 68 65 72 65 0D 0A: `readFile` with utf-8 returns a 12-character string beginning U+FEFF, the decoder with default options returns 11 characters without it, and `{ fatal: true, ignoreBOM: true }` returns 12 characters whose re-encoding is byte-identical to the file. The fix is `ignoreBOM: true`, which restores what the tool did before my change rather than inventing new behaviour. * The offset. A BOM is three bytes that decode to one character, so counting through the decoded string reported a bad byte three positions early (measured: offset 3 where the offending byte is at 6), and a U+FFFD that the file itself contains is indistinguishable in the decoder's output from a replacement the decoder produced, so a real one earlier in the file was read as the failure position (measured: offset 1 where the offending byte is at 5). The offset is now walked in the original bytes, stepping over a replacement only when those bytes really are EF BB BD. Both are checked against byte positions read off a hex dump rather than against anything derived from the decoded string: BOM plus invalid -> 4, literal U+FFFD plus invalid -> 4, both together -> 6, bad continuation -> 1, truncated sequence -> 1, invalid restricted continuation (ED A0 80) -> 1, multibyte then invalid -> 2, and the two rows already in the spec -> 3 and -> 1. Nine cases, all agreeing with the tool's own output on the pinned interpreter. Verification, pinned node v22.23.4 with stdio on file descriptors: - src/core/tools/__tests__/applyDiffTool.failureReporting.spec.ts: 44 passed (1 file), four of them new. - tsc --noEmit -p tsconfig.json in src: 0 errors. - eslint --prune-suppressions --max-warnings=0: 0 on both files; its rewrite of src/eslint-suppressions.json was reverted, so no suppression churn rides along. - prettier --check clean on both, formatted before measuring. - The merge candidate (this head onto upstream/main e487cd7, which is what the pull_request CI actually checks out) is run locally before the push, because main's own applyDiffTool.outsideWorkspace.spec.ts runs against this file there. Gate: 355 of 500 counted executable lines at base 73931fe with base...head and --unified=0 on the formatted tree, against 347 at d5c31a1 - a +8-line move, measured with v22.23.4. Spec rows cost nothing: the four new rows are in __tests__, which the gate excludes.
1 parent d5c31a1 commit eb15d53

2 files changed

Lines changed: 67 additions & 9 deletions

File tree

‎src/core/tools/ApplyDiffTool.ts‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -91,10 +91,28 @@ export class ApplyDiffTool extends BaseTool<"apply_diff"> {
9191
const originalBytes = Buffer.isBuffer(raw) ? raw : Buffer.from(raw, "utf-8")
9292
let originalContent: string
9393
try {
94-
originalContent = new TextDecoder("utf-8", { fatal: true }).decode(originalBytes)
94+
originalContent = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(originalBytes)
9595
} catch {
96-
const lossy = new TextDecoder("utf-8").decode(originalBytes)
97-
const undecodableByte = new TextEncoder().encode(lossy.slice(0, lossy.indexOf("\uFFFD"))).length
96+
const lossy = new TextDecoder("utf-8", { ignoreBOM: true }).decode(originalBytes)
97+
// The offset is counted back through the bytes, not read off the decoded string. Two things
98+
// make the string lie: a byte-order mark is three bytes that decode to one character, so a
99+
// count through the string is three short on every BOM file, and a U+FFFD that the file itself
100+
// contains is indistinguishable in the output from a replacement the decoder produced, so the
101+
// first one of those is read as the failure position too. Each replacement is therefore located
102+
// in the original bytes and only stepped over when those bytes really are the encoded form of
103+
// U+FFFD, which is the one case where moving past it is correct.
104+
let undecodableByte = 0
105+
for (const prefix of lossy.split("\uFFFD")) {
106+
undecodableByte += Buffer.byteLength(prefix, "utf-8")
107+
if (
108+
originalBytes[undecodableByte] !== 0xef ||
109+
originalBytes[undecodableByte + 1] !== 0xbf ||
110+
originalBytes[undecodableByte + 2] !== 0xbd
111+
) {
112+
break
113+
}
114+
undecodableByte += 3
115+
}
98116
task.consecutiveMistakeCount++
99117
task.recordToolError("apply_diff")
100118
const formattedError = `File ${absolutePath} is not valid UTF-8 (first undecodable byte at offset ${undecodableByte}).\n\n<error_details>\napply_diff decodes the file, applies the diff to the decoded text, and saves the result back. Bytes that cannot be decoded would have been saved back as U+FFFD alongside every byte the diff did not touch, so the edit was refused and nothing was written. Re-read the file and convert it to UTF-8 before applying the diff again.\n</error_details>`

‎src/core/tools/__tests__/applyDiffTool.failureReporting.spec.ts‎

Lines changed: 46 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ describe("ApplyDiffTool failure reporting", () => {
154154
}
155155

156156
describe("bytes the decoder cannot read", () => {
157-
async function runOnBytes(bytes: string | Buffer): Promise<string> {
157+
async function runOnBytes(bytes: string | Buffer): Promise<{ output: string; appliedTo: unknown }> {
158158
mockedReadFile.mockResolvedValue(bytes)
159159
const applyDiff = vi.fn().mockResolvedValue({ success: true, content: "unused" })
160160
mockTask.diffStrategy = { getName: () => "test", applyDiff }
@@ -166,15 +166,18 @@ describe("ApplyDiffTool failure reporting", () => {
166166
pushedResult = result
167167
}),
168168
})
169-
return String(pushedResult ?? "")
169+
return {
170+
output: String(pushedResult ?? ""),
171+
appliedTo: applyDiff.mock.calls[0] ? applyDiff.mock.calls[0][0] : undefined,
172+
}
170173
}
171174

172175
// A byte the UTF-8 decoder rejects sits at index 3; everything around it is plain ASCII, so the
173176
// offset the tool reports has to be that index and nothing else.
174177
const UNDECODABLE = Buffer.from([0x68, 0x69, 0x20, 0xff, 0x61, 0x0a])
175178

176179
it("refuses the edit and names the file and the first undecodable byte", async () => {
177-
const output = await runOnBytes(UNDECODABLE)
180+
const { output } = await runOnBytes(UNDECODABLE)
178181

179182
expect(output).toContain("is not valid UTF-8")
180183
expect(output).toContain("first undecodable byte at offset 3")
@@ -196,7 +199,7 @@ describe("ApplyDiffTool failure reporting", () => {
196199
})
197200

198201
it("still edits a file whose bytes are valid multi-byte UTF-8", async () => {
199-
const output = await runOnBytes(Buffer.from("héllo 中文 ✅\n", "utf8"))
202+
const { output } = await runOnBytes(Buffer.from("héllo 中文 ✅\n", "utf8"))
200203

201204
expect(output).not.toContain("is not valid UTF-8")
202205
expect(output).toContain("Changes successfully applied.")
@@ -209,19 +212,56 @@ describe("ApplyDiffTool failure reporting", () => {
209212
// file (the merge ref of 1910 onto main failed 9 of its rows under the strict read), and that spec
210213
// is main's, so this branch cannot edit it. This row is the pin: it says which side gives way.
211214
it("edits a file a caller already decoded, which is what the encoding overload hands back", async () => {
212-
const output = await runOnBytes("line one\nline two\nline three\n")
215+
const { output } = await runOnBytes("line one\nline two\nline three\n")
213216

214217
expect(output).toContain("Changes successfully applied.")
215218
expect(mockTask.consecutiveMistakeCount).toBe(0)
216219
expect(mockTask.diffViewProvider.saveChanges).toHaveBeenCalled()
217220
})
218221

219222
it("reports the first undecodable byte when invalid bytes repeat", async () => {
220-
const output = await runOnBytes(Buffer.from([0x41, 0xc3, 0x28, 0x42, 0xfe, 0x43, 0xff]))
223+
const { output } = await runOnBytes(Buffer.from([0x41, 0xc3, 0x28, 0x42, 0xfe, 0x43, 0xff]))
221224

222225
expect(output).toContain("first undecodable byte at offset 1")
223226
expect(output).not.toContain("offset 4")
224227
})
228+
229+
// A byte-order mark is three bytes that decode to one character, and a U+FFFD written in the file is
230+
// indistinguishable in the decoded output from a replacement the decoder produced. Both make a
231+
// count through the decoded string report the wrong byte, so the offsets below are asserted
232+
// against the byte positions a hex dump shows, not against anything derived from the string.
233+
it("counts a byte-order mark as the three bytes it is", async () => {
234+
const { output } = await runOnBytes(Buffer.from([0xef, 0xbb, 0xbf, 0x41, 0xff]))
235+
236+
expect(output).toContain("first undecodable byte at offset 4")
237+
expect(output).not.toContain("offset 1")
238+
})
239+
240+
it("does not mistake a literal replacement character for an undecodable byte", async () => {
241+
const { output } = await runOnBytes(Buffer.from([0xef, 0xbf, 0xbd, 0x41, 0xff]))
242+
243+
expect(output).toContain("first undecodable byte at offset 4")
244+
expect(output).not.toContain("offset 0")
245+
})
246+
247+
it("counts past a byte-order mark and a literal replacement character together", async () => {
248+
const { output } = await runOnBytes(Buffer.from([0xef, 0xbb, 0xbf, 0xef, 0xbf, 0xbd, 0xff]))
249+
250+
expect(output).toContain("first undecodable byte at offset 6")
251+
})
252+
253+
// The BOM survives into the text the diff is applied to and the approval baseline is built from,
254+
// which is what fs.readFile with a utf-8 encoding used to hand over: the decoded string plus the
255+
// BOM character, and re-encoding it reproduces the file's bytes exactly.
256+
it("keeps a byte-order mark in the content the diff and the approval are built from", async () => {
257+
const bomFile = Buffer.from("\uFEFFhi\r\nthere\r\n", "utf8")
258+
const { output, appliedTo } = await runOnBytes(bomFile)
259+
260+
expect(output).toContain("Changes successfully applied.")
261+
expect(typeof appliedTo).toBe("string")
262+
expect(String(appliedTo).startsWith("\uFEFF")).toBe(true)
263+
expect(Buffer.from(String(appliedTo), "utf8").equals(bomFile)).toBe(true)
264+
})
225265
})
226266

227267
describe("failure accumulation and rendering", () => {

0 commit comments

Comments
 (0)