Read tool is not byte-faithful: invalid UTF-8 silently replaced, truncated multi-byte accepted as text #57837

Activity

1. opencode-agent commented on Oct 21, 2026

   

   opencode-agentBot

   on Oct 21, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the clear report. I reproduced this on v2.1.2 and on the latest dev (002d7bd81), using a simulated model:

   • invalid.txt holds const x = 1 followed by a comment line containing the bytes d6 d0 ce c4 ff (GBK text plus a stray 0xff).
   • truncated.txt holds hello world\n followed by e4 b8, a 3-byte UTF-8 character cut off after two bytes.

   The read tool rejected neither file as binary. Both came back as normal text with U+FFFD in place of the original bytes:

   Read file invalid.txt, lines 1-2
   1: const x = 1
   2: // �����
   

   Read file truncated.txt, lines 1-2
   1: hello world
   2: �
   

   So the model sees content that differs from the file on disk, as you describe. The script below reproduces it, and the recording shows the run.

   Related:

   • fix(read): treat invalid UTF-8 as binary instead of silently replacing bytes #57833: your PR for this.
   • edit and apply_patch silently corrupt non-UTF-8 files (U+FFFD replacement) #45924: the same lossy decoding in edit/apply_patch, which can corrupt files when they're written back.
   • [bug] Read and Edit tools handle invalid UTF-8 inconsistently: one crashes, the other silently corrupts #33068: an earlier (closed) report on Read and Edit handling invalid UTF-8 differently.

   recording.mp4

   Reproduction script (drive.ts)

   // Repro for #57837: the read tool returns lossy text (U+FFFD) for files containing invalid UTF-8.
   import { Effect, Stream } from "effect"
   import { Llm, OpenCodeDriver } from "opencode-drive"
   
   const enc = (s: string) => new TextEncoder().encode(s)
   const cat = (...parts: Uint8Array[]) => {
     const out = new Uint8Array(parts.reduce((n, p) => n + p.length, 0))
     let o = 0
     for (const p of parts) (out.set(p, o), (o += p.length))
     return out
   }
   // GBK bytes for 中文 (invalid UTF-8) plus a stray 0xff
   const invalid = cat(enc("const x = 1\n// "), new Uint8Array([0xd6, 0xd0, 0xce, 0xc4, 0xff]), enc("\n"))
   // valid text ending in a truncated 3-byte UTF-8 sequence (first 2 bytes of 中 = e4 b8 ad)
   const truncated = cat(enc("hello world\n"), new Uint8Array([0xe4, 0xb8]))
   
   export default OpenCodeDriver.use(
     {
       ...(process.env.OC_DEV ? { opencode: { dev: process.env.OC_DEV } } : {}),
       project: { git: true, files: { "invalid.txt": invalid, "truncated.txt": truncated } },
       tui: { recording: true },
     },
     ({ ui, llm }) =>
       Effect.gen(function* () {
         let called = false
         yield* llm.serve((request: any, index) => {
           console.log("LLM request", index)
           if (JSON.stringify(request).includes("You name conversations")) return Stream.make(Llm.text("Read test"))
           if (!called && (called = true))
             return Stream.make(
               Llm.toolCall({ index: 0, id: "call_invalid", name: "read", input: { path: "invalid.txt" } }),
               Llm.toolCall({ index: 1, id: "call_trunc", name: "read", input: { path: "truncated.txt" } }),
               Llm.finish("tool-calls"),
             )
           const body = JSON.stringify(request)
           const fffd = (body.match(/\uFFFD|\\ufffd/gi) || []).length
           const found: any[] = []
           const walk = (v: any) => {
             if (Array.isArray(v)) return v.forEach(walk)
             if (v && typeof v === "object") {
               if (v.role === "tool") found.push(v)
               Object.values(v).forEach(walk)
             }
           }
           walk(request)
           const msgs = found
           for (const m of msgs) console.log("TOOL RESULT", m.tool_call_id, JSON.stringify(m.content))
           console.log("U+FFFD occurrences in model request:", fffd)
           return Stream.make(Llm.text(`Tool results contained ${fffd} U+FFFD replacement characters.`))
         })
         yield* ui.submit("Read invalid.txt and truncated.txt")
         yield* ui.waitFor("replacement characters", { timeout: 60000 })
         yield* Effect.sleep(1500)
         yield* ui.screenshot("result")
       }),
   )

   Run it with opencode-drive: opencode-drive run ./drive.ts. It drives opencode from your PATH; set OC_DEV to the path of an OpenCode checkout to run that from source instead.
2. opencode-agent

   added

   reproduced

   on Oct 21, 2026