bug: Read truncation can split Unicode surrogate pairs #47399

Activity

1. github-actions

   assigned

   neriousy

   on Sep 5, 2026
2. opencodejr commented on Oct 28, 2026

   

   opencodejrBot

   on Oct 28, 2026

   Contributor

   More actions

   Thanks for the precise report. I reproduced this on the latest dev (35a4a68) with the v2 Core reader. Reading a file containing "a".repeat(1999) + "😀tail\nnext" with { limit: 1 } returns a page where code unit 1999 is a lone \uD83D, the first half of the emoji, directly before the ... (line truncated to 2000 chars) marker. The model-facing text has the same lone surrogate, and it turns into U+FFFD when encoded as UTF-8. The cut happens in clampLine in packages/core/src/tool/read-filesystem.ts, which slices by UTF-16 code units. The script below reproduces it against a source checkout.

   I only checked the v2 Core path. The legacy Read tool belongs to v1 and wasn't re-tested.

   Reproduction script (repro.sh)

   #!/usr/bin/env bash
   # Reproduces #47399 against an opencode checkout (default /workspace/repo) with `bun install` done.
   set -euo pipefail
   REPO="${1:-/workspace/repo}"
   TEST="$REPO/packages/core/test/repro-47399.test.ts"
   cat > "$TEST" <<'TS'
   import { describe, expect } from "bun:test"
   import path from "path"
   import { Environment } from "@opencode/core/environment/index"
   import { AbsolutePath } from "@opencode/core/schema"
   import { ReadTool } from "@opencode/core/tool/plugin/read"
   import { ReadToolFileSystem } from "@opencode/core/tool/read-filesystem"
   import { CrossSpawnSpawner } from "@opencode/util/cross-spawn-spawner"
   import { LayerNodePlatform } from "@opencode/util/effect/app-node-platform"
   import { LayerNode } from "@opencode/util/effect/layer-node"
   import { Effect, FileSystem } from "effect"
   import { ChildProcessSpawner } from "effect/process"
   import { testEffect } from "./lib/effect"
   
   const it = testEffect(LayerNode.compile(LayerNode.group([CrossSpawnSpawner.node, LayerNodePlatform.filesystem])))
   const lone = (s: string) => /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/.test(s)
   
   describe("#47399", () => {
     it.live("line truncation does not split a surrogate pair", () =>
       Effect.gen(function* () {
         const files = yield* FileSystem.FileSystem
         const env = Environment.makeFiles(Environment.makeLocalDriver(yield* ChildProcessSpawner.ChildProcessSpawner))
         const dir = yield* files.makeTempDirectoryScoped()
         const file = AbsolutePath.make(path.join(dir, "unicode-boundary.txt"))
         yield* files.writeFileString(file, "a".repeat(1999) + "😀tail\nnext")
         const result: any = yield* ReadToolFileSystem.read(env, file, "unicode-boundary.txt", { limit: 1 })
         const model = ReadTool.toModelContent("unicode-boundary.txt", undefined, result)
         const line = result.content as string
         console.log("type:", result.type, "content length:", line.length)
         console.log("char codes 1997..2001:", [...line.slice(1997, 2001)].map((c) => c.charCodeAt(0).toString(16)))
         console.log("lone surrogate in content:", lone(line), "| in model text:", lone(model))
         console.log("UTF-8 round trip contains U+FFFD:", Buffer.from(line, "utf8").toString("utf8").includes("\uFFFD"))
         expect(lone(line)).toBe(false)
         expect(lone(model)).toBe(false)
       }),
     )
   })
   TS
   cd "$REPO/packages/core"
   trap 'rm -f "$TEST"' EXIT
   bun test test/repro-47399.test.ts

   Run it: bash repro.sh
3. opencodejr

   added

   reproduced

   on Oct 28, 2026
4. Tarquinen

   closed this as completedin #62575on Nov 10, 2026