Encoding issue: non-UTF-8 encoded files on Windows [I find a solution] #30187

Description

On Windows with Chinese locale (GBK/CP936 system code page), ilesystem_read_text_file produces garbled output when reading files encoded in GBK, GB2312, or other legacy CJK encodings.

Example — reading a GBK markdown file:
filesystem_read_text_file → ���� GBK ��������IJ����ļ���Hello World��

The tool always assumes UTF-8 decoding with no encoding auto-detection fallback.

Root cause

ilesystem_read_text_file reads bytes and decodes as UTF-8 unconditionally. When the file uses GBK (the default encoding on Chinese Windows for legacy applications), every multi-byte CJK character is misinterpreted as UTF-8 replacement characters (U+FFFD).

Expected behavior

Should detect encoding transparently — same as Notepad++, VS Code, and Obsidian all do on the same files:

1. Check BOM first (UTF-8 BOM, UTF-16 LE/BE)
2. Validate if already valid UTF-8
3. Fall back to statistical chardet detection for legacy encodings (GBK, Shift-JIS, EUC-KR, etc.)
4. Always deliver decoded text as UTF-8 to the LLM

Proposed solution

Integrate jschardet (Node.js port of Mozilla Universal Charset Detector, ~50KB, MIT) into the file read path — same algorithm used by Firefox/VS Code/Notepad++ for decades:

` ypescript
// Simplified example
import jschardet from "jschardet";

function readAndDecode(buffer: Buffer): string {
// 1. BOM check
if (buffer[0] === 0xEF && buffer[1] === 0xBB && buffer[2] === 0xBF)
return buffer.toString("utf-8", 3);
if (buffer[0] === 0xFF && buffer[1] === 0xFE)
return buffer.toString("utf-16le", 2);
if (buffer[0] === 0xFE && buffer[1] === 0xFF)
return buffer.toString("utf-16be", 2);

// 2. Try pure UTF-8 first
try {
const text = buffer.toString("utf-8");
if (text.includes("\uFFFD")) throw new Error("lossy decode");
return text;
} catch {
// 3. Statistical detection
const result = jschardet.detect(buffer);
const enc = result?.encoding?.toLowerCase() || "utf-8";
return iconv.decode(buffer, enc);
}
}
`

Environment

• OS: Windows 11, zh-CN locale
• Terminal: Windows Terminal
• OpenCode version: 1.15.12

Plugins

opencode-ignore, [email protected]

Steps to reproduce

1. On Windows with zh-CN locale, create a file saved as GBK encoding containing Chinese text like 你好世界
2. Ask OpenCode to read or edit this file
3. Observe garbled output in the tool response

Related issues

• encoding problem #7933 — Same root cause for EUC-KR (Korean files)
• fix: PowerShell output encoding for non-ASCII characters on Windows #23636 — PowerShell output encoding (terminal layer, different issue but same Windows GBK context)
• Icons/Unicode characters cause file editing failures #2904 — �dit tool fails with Unicode multi-byte characters
• Fix CJK character display issues in Windows terminals #10491 / 命令行模式下中文显示突然全部变成乱码 #9793 / OpenCode TUI Chinese Text Display Issue #7891 — CJK display in TUI (terminal side, partially fixed by fix(pty): Add UTF-8 encoding defaults for Windows PTY #11459, but file tools were never addressed)