edit and apply_patch silently corrupt non-UTF-8 files (U+FFFD replacement) #45924

Activity

1. github-actions

   assigned

   neriousy

   on Aug 28, 2026
2. ekoleszar commented on Sep 1, 2026

   

   ekoleszar

   on Sep 1, 2026

   Last edited by ekoleszar

   More actions

   Occurrence: ISO-8859-1 Java project

   Same root cause, confirmed in an ISO-8859-1 (Latin-1) legacy Java project where the build compiles with javac -encoding ISO-8859-1.

   Observed behavior: the edit tool replaced Spanish non-ASCII bytes (0xF1 = ñ, 0xE1 = á, etc.) with U+FFFD (0xEF 0xBF 0xBD) while editing ASCII regions of the same file. The corruption is silent: no error, and the tool reports success.

   Why it hurt: because javac -encoding ISO-8859-1 accepts every 0x00-0xFF byte, the corrupted file still compiles — but the .class constant pool now contains mojibake strings (e.g. 'más' instead of 'más'). The bug only shows up at runtime, or when inspecting the binary .class.

   Environment: OpenCode 1.18.25, Linux, Java source files with Spanish strings encoded in ISO-8859-1.

   Recovery used: restored files from git (git checkout -- <file>) and re-applied the same edits via Python 'rb'/'wb'` raw-byte replacement, which preserves encoding byte-for-byte.

   Suggested scope for the fix: match the read tool behavior — reject non-UTF-8 files with a fatal decoder before edit/apply_patch write back, OR preserve detected original encoding on write. At minimum, refuse to write when the file is not valid UTF-8, instead of silently destroying bytes.
3. MasEne1 commented on Sep 19, 2026

   

   MasEne1

   on Sep 19, 2026

   More actions

   I have a working implementation of the second option mentioned above (preserve the detected original encoding on write), and would like to offer it as a complement to #45925.

   #45925 protects the core (v2) tools by rejecting non-UTF-8 files — the right minimal fix there. This covers the v1 tools in packages/opencode (edit / write / read / apply_patch + the patch engine), which currently still have the same bug, and goes one step further: instead of refusing to touch non-UTF-8 files, it detects the encoding per read (BOM → UTF-8 validity → configurable fallback) and writes back in the same encoding, so legacy files stay editable:

   • New config file_encoding (default "utf-8" → default behavior unchanged), e.g. "gbk", "shift_jis"
   • Files with BOM (UTF-8 / UTF-16LE / UTF-16BE) work without any configuration
   • Other encodings via lazily-imported iconv-lite (zero cost on the UTF-8 path)
   • Byte-level round-trip tests: GBK edit/write/read, UTF-16 BOM, UTF-8 BOM preserved, file left byte-identical when an edit does not match

   Happy to scope it down or rebase however maintainers prefer (e.g. land after #45925). Opening a PR referencing this issue.
4. jlgascon-hiberuscom commented on Oct 20, 2026

   

   jlgascon-hiberuscom

   on Oct 20, 2026

   More actions

   Confirmed on OpenCode 2.0.32 / Windows 11 with Windows-1252 (cp1252) + CRLF Informix 4GL files.

   A one-space edit corrupted all 745 existing non-ASCII characters:

   Before: 421899 bytes, 0 U+FFFD
   After: 423390 bytes, 745 U+FFFD

   The +1491 bytes match exactly: 745 × 2 extra bytes from U+FFFD + the single space added.

   So this is still reproducible in 2.0.32 with cp1252 files.
5. jlgascon-hiberuscom

   marked Edit/Write corrupt Windows-1252 (cp1252) files by rewriting them as UTF-8 #57308 as a duplicate of this issue on Oct 20, 2026
6. opencodejr commented on Oct 28, 2026

   

   opencodejrBot

   on Oct 28, 2026

   Contributor

   More actions

   Thanks for the clear report. I reproduced this on 2.2.0 and it still happens on the latest dev (506ac20).

   Setup: a file containing const x = 1\n// followed by the GBK bytes d6 d0 ce c4, and a simulated model.

   • edit (const x = 1 → const y = 1): the tool reports success, and the diff shows // ����. od -An -tx1 on the file then shows … 2f 2f 20 ef bf bd ef bf bd ef bf bd ef bf bd 0a, so all four GBK bytes were replaced with U+FFFD.
   • patch (offered instead of edit for gpt- models): the same update corrupts the file the same way on 2.2.0.

   The code paths in the issue have moved. On current dev, edit, patch and write all read files through FileMutation.readText, which decodes with a non-fatal TextDecoder in packages/util/src/bom.ts.

   The recording and screenshots below show it, and the script below reproduces it.

   Related: #49600 and #57308 were closed as duplicates of this issue. #45925 (fatal-decoder fix, closed without merging) and #58113 (open PR adding non-UTF-8 encoding support) are related PRs.

   recording.mp4

   

   

   

   Reproduction script (drive.ts)

   // Repro for anomalyco/opencode#45924: edit/patch silently replace non-UTF-8 bytes with U+FFFD.
   // TOOL=edit (default) uses a non-"gpt-" model id so the `edit` tool is offered;
   // TOOL=patch uses a "gpt-" model id so the `patch` tool is offered.
   import { Effect, Stream } from "effect"
   import { Llm, OpenCodeDriver } from "opencode-drive"
   
   const tool = process.env.TOOL === "patch" ? "patch" : "edit"
   const modelID = tool === "patch" ? "gpt-sim-model" : "sim-model"
   
   // "const x = 1\n// " + GBK bytes for 中文 (d6 d0 ce c4) + "\n"
   const bytes = new Uint8Array([...new TextEncoder().encode("const x = 1\n// "), 0xd6, 0xd0, 0xce, 0xc4, 0x0a])
   
   const input: Record<string, string> =
     tool === "edit"
       ? { path: "gbk.ts", oldString: "const x = 1", newString: "const y = 1" }
       : { patchText: "*** Begin Patch\n*** Update File: gbk.ts\n@@\n-const x = 1\n+const y = 1\n*** End Patch" }
   
   export default OpenCodeDriver.use(
     {
       ...(process.env.OC_DEV ? { opencode: { dev: process.env.OC_DEV } } : {}),
       config: {
         model: `simulation/${modelID}`,
         providers: {
           simulation: {
             models: {
               [modelID]: {
                 name: "Simulated Model",
                 capabilities: { tools: true, input: ["text"], output: ["text"] },
                 limit: { context: 128000, output: 16000 },
               },
             },
           },
         },
       } as any,
       project: { git: true, files: { "gbk.ts": bytes } },
       tui: { recording: true },
     },
     ({ ui, llm }) =>
       Effect.gen(function* () {
         // Answer by conversation state; requests without tools (e.g. title generation) get plain text.
         yield* llm.serve((request: any) => {
           const body = JSON.stringify(request?.body ?? request)
           const hasTools = Array.isArray(request?.body?.tools) && request.body.tools.length > 0
           if (!hasTools) return Stream.make(Llm.text("GBK edit repro"), Llm.finish())
           if (body.includes("call_od")) return Stream.make(Llm.text("DONE: checked bytes after the change."), Llm.finish())
           if (body.includes("call_tool"))
             return Stream.make(
               Llm.toolCall({ index: 0, id: "call_od", name: "shell", input: { command: "od -An -tx1 gbk.ts", description: "hexdump" } }),
               Llm.finish("tool-calls"),
             )
           return Stream.make(Llm.toolCall({ index: 0, id: "call_tool", name: tool, input }), Llm.finish("tool-calls"))
         })
         yield* ui.submit(`Change x to y in gbk.ts (original bytes: ... 2f 2f 20 d6 d0 ce c4 0a)`)
         yield* ui.waitFor("DONE: checked bytes after the change.", { timeout: 60_000 })
         yield* Effect.sleep("1 second")
         yield* ui.screenshot(`after-${tool}`)
       }),
   )

   Run it with opencode-drive: opencode-drive run ./drive.ts. It drives opencode from your PATH; set OC_DEV to the path of an OpenCode checkout to run that from source instead.
7. opencodejr

   added

   reproduced

   severity:high

   on Oct 28, 2026
8. easyteacher commented on Nov 8, 2026

   

   easyteacher

   on Nov 8, 2026

   More actions

   Additional evidence on 2.2.6 (Windows) — read is no longer a safe guard

   Re-confirming the corruption, plus one point that changes the premise of the report: on 2.2.6 the read tool no longer rejects non-UTF-8 files, so an agent has no way to notice the encoding before edit destroys it.

   Environment

   • opencode 2.2.6 (release)
   • Windows 11, Windows_NT 10.0.26300
   • Shell: C:\WINDOWS\system32\cmd.exe, commands run through PowerShell
   • No encoding-related plugins or config

   Test matrix

   Files created with .NET WriteAllText; content line1 hello 中文测试\r\nline2 end\r\n.

   File encoding	read	edit	write
   UTF-8, no BOM	correct	correct, stays BOM-less	UTF-8 no BOM, always
   UTF-8, BOM EF BB BF	correct, BOM hidden	correct, BOM preserved	n/a
   UTF-16LE, BOM FF FE	Cannot read binary file	Could not find oldString, file untouched	n/a
   GBK (d6 d0 ce c4 = 中文)	mojibake, no error	silent corruption	n/a

   1. read silently returns U+FFFD instead of failing

   The report says read already rejects this via MalformedUtf8Error. That is not what 2.2.6 does:

   file bytes: 61 62 63 20 d6 d0 ce c4 20 65 6e 64 0a
   read result: "abc ���� end"
   

   It returns successfully with replacement characters — no MalformedUtf8Error, no "file is not valid UTF-8". This matters beyond cosmetics: the model sees plausible text, has no signal that the file is non-UTF-8, and proceeds to edit it.

   2. edit then destroys the remaining bytes

   edit with oldString: "line2 end" → newString: "line2 changed" on the same GBK file:

   before: 6c 69 6e 65 31 20 68 65 6c 6c 6f 20 d6 d0 ce c4 b2 e2 ca d4 0d 0a 6c 69 6e 65 32 20 65 6e 64 0d 0a
   after:  6c 69 6e 65 31 20 68 65 6c 6c 6f 20 ef bf bd ef bf bd ef bf bd c4 b2 ef bf bd ef bf bd ef bf bd 0d 0a 6c 69 6e 65 32 20 63 68 61 6e 67 65 64 0d 0a
   

   The tool reports Edited ... (1 replacement). All 8 GBK bytes are replaced by U+FFFD.

   3. UTF-16 with BOM is rejected as "binary", not decoded

   A FF FE BOM file hits the NUL-byte heuristic and returns Cannot read binary file, so BOM-based encoding detection is not active for UTF-16. Oddly this makes UTF-16 the safe case: edit fails with Could not find oldString and leaves the file byte-identical.

   Suggestion

   Make the failure loud at the first touch, in read as well as edit/patch/write:

   • decode with a fatal UTF-8 decoder in read, and fail with an explicit File is not valid UTF-8 (the read path currently loses this),
   • apply the same fatal decode before any write in edit/patch/write, so a non-UTF-8 file is refused rather than rewritten,
   • keep BOM handling (it works today for UTF-8 and is preserved by edit),
   • optionally add an explicit encoding option or detect the BOM for UTF-16.

   Rejecting is strictly better than the current state, because a model that is told "not UTF-8" will convert the file with a byte-preserving tool, while the current silent U+FFFD replacement is unrecoverable without git.