feat(cli): preserve original text encoding when editing files - #8587

…non-Latin files

ReadTool now uses Encoding.detect + iconv-lite to decode files in any
supported encoding (Shift-JIS, GB2312, Big5, EUC-KR, etc.) instead of
hardcoding UTF-8. The binary detection heuristic now checks encoding
first so UTF-16 files with null bytes and CJK encoded files are no
longer falsely rejected as binary.

…etection

Drop manual BOM parsing, UTF-32, and UTF-16-without-BOM detection: rely on
TextDecoder + jschardet + iconv-lite which already handle BOM round-tripping
for UTF-16 LE/BE. Move the encoding helpers and Effect wrappers into
packages/opencode/src/kilocode/ so shared tool files only carry targeted
kilocode_change markers.

Exercise Read/Write/Edit/ApplyPatch tools directly against files in
UTF-8, UTF-16 LE/BE (with BOM), Shift_JIS, EUC-JP, GB2312, Big5, EUC-KR,
Windows-1251, and KOI8-R to verify the tools decode input correctly and
round-trip bytes back to the original encoding. Also ensure UTF-16
BOMs survive the write pipeline.

This reverts commit 227af07.

This reverts commit ce80c09.

- Restore the streaming createInterface loop in the `lines` helper so the
  diff against main stays small; decode bytes with iconv-lite up front and
  feed the resulting text through Readable.from.
- Drop 'new file' kilocode_change markers from files under kilocode/
  directories (those paths are exempt from the annotation check).
- Tighten the changeset copy to sound like a user-facing release note.

- Wire Readable.from through a named `stream` variable so the `lines`
  helper can destroy it in the finally block, matching the previous
  createReadStream pattern.
- Restore the 'Create parent directories', 'Handle file move', 'Regular
  update', 'Read original file content', and 'For delete, we need to read
  the current content' comments that were dropped when switching to
  Encoding.write / Encoding.read. The explicit fs.mkdir calls are no
  longer needed because Encoding.write mkdirs recursively, but the intent
  comments still apply.

…ation

- Encoding.detect: lowercase-compare UTF-32 rejection so jschardet's
  uppercase 'UTF-32'/'UTF-32BE'/'UTF-32LE' labels no longer slip through
  (iconv-lite happens to have utf-32 codecs that would then be accepted).
- Encoding.read: drop redundant Buffer.from wrap around readFile's
  already-Buffer return value.
- EncodedIO: switch to Effect.tryPromise so I/O failures surface as
  typed errors that .pipe(Effect.catch(...)) can recover from; the
  apply_patch delete branch relies on this to translate read errors
  into 'apply_patch verification failed'.
- Shrink kilocode_change blocks in edit.ts, write.ts, apply_patch.ts,
  patch/index.ts, and read.ts to per-line inline markers where each
  block was only wrapping 1-3 changed lines.
- Collapse the 'if (exists) { let contentOld; let encoding; ... }'
  blocks in edit.ts and write.ts into a single ternary + destructure
  that keeps all three lines self-contained and each independently
  annotated.
- Remove the unused 'latin1' sample and fix the 'providEncoded' typo
  in the encoding integration test file.
- Strengthen the apply_patch Shift_JIS test to assert exact bytes
  instead of the weak 'does not contain these two UTF-8 bytes' check.
- Drop the stray .kilo/ and .kilocode/ lockfiles that were re-added
  by an upstream 'wip' commit.

Per review: CJK and legacy single-byte encodings already passed the old
control-char heuristic (their bytes are all >= 0x80, never NUL), so the
full Encoding.detect pass in isBinaryFile was solving a non-problem for
them. The only realistic regression in the old heuristic is UTF-16 with
BOM, where the second byte of every ASCII character is 0x00 and fires
the NUL-byte early-return.

Replace the detect call with a 2-byte BOM check via a new
Encoding.hasUtf16Bom helper. Also reword the move/add inline markers in
patch/index.ts to note that Encoding.write handles mkdir.

…ation

Main moved the Patch namespace out of packages/opencode/src/patch/index.ts
into patch/patch.ts and switched tool imports to the new
@opencode-ai/shared/filesystem and ../../src/tool barrels. Reapply the
encoding-aware read/write changes on top of the refactored patch.ts, and
update the integration test imports to match the new module layout.

- Drop the dead 'encoding = pre.encoding' initial value that was only
  read if deriveNewContentsFromChunks threw, and that code path
  returns before encoding is ever used.
- Inline pre.text into the read line so we don't bind a 'pre' local
  just to destructure it once.