You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Root string starting with U+FEFF is emitted unquoted and loses its first character on decode #339
When the root value is a string whose first character is U+FEFF, the encoder emits it unquoted. The decoder then treats that leading U+FEFF as a byte-order mark and strips it (SPEC §12), so decode(encode(x)) no longer equals x. In the worst case the whole value disappears.
Verified on main (f151a5d) by running the source directly, and on the published 4.1.1 package.
Encoded output for each is the raw string, e.g. encode('8') returns "8" with no quotes.
Only the root primitive position is affected. Keys starting with U+FEFF are already quoted by encodeKey (non-ASCII fails the unquoted-key pattern), and inline array values, tabular cells, list items, and object field values never start the document:
SPEC §12: "A U+FEFF anywhere else is content. Encoders MUST NOT emit one." A document that begins with U+FEFF has, by the spec's own definition, been emitted with a BOM.
SPEC §2: decode(encode(x)) must equal x under JSON-model equality.
The encoder already knows about this trap: RawString in packages/toon/src/encode/raw-string.ts rejects raw strings that would read as a comment line after a leading BOM is removed. The normal string path in isSafeUnquoted (packages/toon/src/shared/validation.ts) has no equivalent check.
Suggested fix
Two options, either is a one-liner:
In encodeJsonValue (packages/toon/src/encode/encoders.ts), when the root value is a string that starts with U+FEFF, force the quoted form. This keeps the change scoped to the only position where it matters.
Add value.startsWith(BYTE_ORDER_MARK) to isSafeUnquoted. Simpler, but it also quotes such strings in positions where it isn't needed.
Option 1 seems closer to the spirit of "quote only when required" (§7.2). Happy to open a PR for whichever you prefer.
It may also be worth a note in §7.2 of the spec, since the current quoting rules do not mention U+FEFF and any implementation that follows them literally will have the same gap. The Rust port has the identical bug and a fix has been requested there.
@johannschopplich flagging this for you since it touches the quoting rules and probably the spec text as well. Happy to send a PR once you've picked an approach.
Went with option 1 – #343 quotes at the root only, and toon-format/spec#60 adds the case to §7.2 with fixtures, so toon-rust#79 can test against those.
Activity
shreyasbhat0 commentedon Sep 15, 2026
@johannschopplich flagging this for you since it touches the quoting rules and probably the spec text as well. Happy to send a PR once you've picked an approach.
fix(toon): keep BOM-led root strings and guard encode normalization
fix(toon): keep BOM-led root strings and guard encode normalization
fix(toon): keep BOM-led root strings and guard encode normalization
johannschopplich commentedon Oct 1, 2026
Went with option 1 – #343 quotes at the root only, and toon-format/spec#60 adds the case to §7.2 with fixtures, so toon-rust#79 can test against those.