Message renderer corrupts non-ASCII characters (em-dash, Latvian diacritics) to "?" #5070

Activity

1. harijsvucens

   added

   bugSomething isn't workingSomething isn't working

   on Aug 6, 2026
2. geosalonica commented on Aug 18, 2026

   

   geosalonica

   on Aug 18, 2026

   More actions

   Root cause found (Windows): content piped into buzz messages send --content - is ASCII-mangled by Windows PowerShell 5.1 before the CLI ever sees it — and the event is signed after the corruption, so the ?s are stored permanently on the relay.

   TL;DR

   • Windows PowerShell 5.1 re-encodes anything piped to a native executable using $OutputEncoding, whose default is US-ASCII. Every non-ASCII character (Greek, Latvian diacritics, em-dashes — all of them) becomes a literal ? (0x3F), one per character, ASCII passes through untouched.
   • buzz messages send --content - reads stdin (strict UTF-8, validate.rs::read_or_stdin) — but the bytes are already 3f 3f 3f … at that point. The CLI then faithfully signs and submits the corrupted content, so it verifies fine and is stored forever.
   • Passing the same text as an argument (--content "…") is lossless on Windows (argv is UTF-16 end-to-end: PowerShell/cmd → CreateProcessW → Rust args_os()), which is why short/single-line messages work and only multiline/piped ones break. This makes the bug look intermittent or like a regression when it's actually shape-dependent.

   This is not a renderer or relay bug — the whole Rust pipeline (buzz-acp → buzz-cli → relay/Postgres) is UTF-8 clean. It also explains why the em-dash in this report became ?: an ANSI-codepage conversion would have preserved it (0x97 in cp1252); only an ASCII conversion kills it.

   Reproduction (Windows 10, PowerShell 5.1)

   powershell -NoProfile -Command '$OutputEncoding.WebName'
   # us-ascii
   
   # pipe Greek + em-dash into any native exe and hex-dump what arrives:
   powershell -NoProfile -Command '"ΕΛΛΗΝΙΚΑ—δοκιμή" | & od.exe -An -tx1'
   #  3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f 3f   <- 15 chars -> 15 '?'

   The same string piped through Git Bash (printf '%s' '…' | od -An -tx1) arrives as perfect UTF-8.

   How agents end up on the corrupting path

   Two compounding factors, both observed on a real deployment (Buzz Desktop 0.5.16, managed Claude agents on Windows):

   1. crates/buzz-acp/src/base_prompt.md recommends the multiline send pattern printf '…' | buzz messages send --content -. That idiom is safe only in bash. Agents running a PowerShell tool translate it to $body = @'…'@; $body | buzz messages send --content - → corrupted.
   2. On the affected machine, Git Bash failed to resolve the extension-less buzz (only buzz.exe is on PATH; command -v buzz fails while command -v buzz.exe succeeds), so the agent's first attempt via bash died with buzz: command not found and it fell back to the PowerShell pipe. So even agents that prefer the safe path get funneled into the unsafe one.

   Verified from the agent transcripts: identical Greek text sent via --content "…" (argv) arrived intact; the same session's here-string + pipe sends arrived as ?????.

   Suggested fixes

   • Docs/prompt (cheap, high impact): in base_prompt.md (and the bundled buzz-cli skill), state that on Windows content must never be piped into buzz from PowerShell; recommend --content "…" as an argument (argv is Unicode-safe, multiline included), or require $OutputEncoding = [Console]::OutputEncoding = [Text.UTF8Encoding]::new() before any pipe.
   • CLI: add --content-file <path> (read raw bytes as UTF-8 straight from disk, no console/pipe encoding in the loop). This gives agents a shell-agnostic safe path for large/multiline content.
   • CLI (defensive): when stdin content contains runs of ? and zero non-ASCII bytes, print a warning to stderr pointing at this encoding trap (heuristic, but would have made this failure loud instead of silent).
   • Packaging: ship an extension-less buzz sh-shim alongside buzz.exe (npm-style) so MSYS/Git-Bash command -v buzz always resolves — this keeps agents on the UTF-8-clean bash path.

   Workaround for affected users (no rebuild needed)

   1. Create an extension-less wrapper ~/bin/buzz for Git Bash:

      #!/bin/sh
      exec "/c/Users/<you>/AppData/Local/Buzz/buzz.exe" "$@"

   2. Add to your PowerShell profile: $OutputEncoding = [Console]::OutputEncoding = [Text.UTF8Encoding]::new()
   3. Optionally add a standing instruction for your agents (e.g. in ~/.buzz/AGENTS.md below the managed markers, which survives upgrades): never pipe content into buzz from PowerShell; pass --content as an argument or use bash.

   Already-corrupted messages can't be repaired (they're signed events) — re-send after applying the workaround. Confirmed working on the affected deployment.
3. bernhardkaindl

   added a commit that references this issue on Sep 12, 2026

   buzz-dev-mcp: configure PowerShell UTF-8 pipes

   ...

   2a0c1c0