Skip to content

Edit silently corrupts non-UTF-8 files (ISO-8859-1 bytes replaced with U+FFFD) #49600

Closed

Description

@ekoleszar

Summary

The edit tool silently destroys non-UTF-8 bytes. Editing an ASCII-only region of a file that contains ISO-8859-1 bytes elsewhere rewrites those bytes as U+FFFD (EF BF BD), reports success, and leaves phantom diff hunks. This is data loss with no warning.

Minimal reproduction

printf 'const x = 1;\n// caf\xe9 note\n' > sample.txt  # 0xE9 = e-acute in ISO-8859-1
# then use the edit tool: oldString "const x = 1;" -> newString "const x = 2;"

Before (xxd): ... 63 61 66 e9 20 ...
After: ... 63 61 66 ef bf bd 20 ... — the untouched 0xE9 byte became U+FFFD. The tool reported success.

Real-world hit: a 1140-line ISO-8859-1 Java file where I made one logical change came back with ~10 unrelated comment lines rewritten (0xF3 -> EF BF BD). Caught only via git diff review; without that check it would have been committed corrupted.

Expected

Either decode with { fatal: true } and refuse with a "file is not valid UTF-8" error, leaving the file untouched (matching the read tool's MalformedUtf8Error behavior), or preserve the original encoding on write. Never silently replace bytes and report success.

Environment

  • opencode 1.18.30, linux
  • files: ISO-8859-1, both CRLF (real case) and LF (minimal repro)

Related

Same defect class as #28521 (Windows-1252), #7933, #27832, #3342 — all currently closed by the stale bot with no fix landed. Open fix PR #26406 (chardet + iconv-lite) would address this; this report adds a fresh byte-level repro against the current version to validate against.

Activity

  1. github-actions commented on Sep 17, 2026

    @github-actions
    Contributor

    This issue might be a duplicate of an existing issue. Please check:

  2. ekoleszar commented on Sep 17, 2026

    @ekoleszar
    Author

    Duplicate of #45924 — closing in favor of the canonical tracker.

    Keeping the fresh evidence here for the record: reproduced again today on opencode 1.18.30 (linux) with a minimal case — printf a file containing byte 0xE9 (ISO-8859-1 e-acute), ASCII-only edit via the edit tool, untouched 0xE9 byte came back as EF BF BD (U+FFFD) while the tool reported success. Same-day real-world hit in an ISO-8859-1 Java file (0xF3 -> U+FFFD, ~10 phantom hunks, caught only via git diff review). So this still reproduces on current builds; validating fix PR #45925 against this case would be appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions