Skip to content

Root string starting with U+FEFF is emitted unquoted and loses its first character on decode #339

Closed

Description

@shreyasbhat0

Summary

When the root value is a string whose first character is U+FEFF, the encoder emits it unquoted. The decoder then treats that leading U+FEFF as a byte-order mark and strips it (SPEC §12), so decode(encode(x)) no longer equals x. In the worst case the whole value disappears.

Verified on main (f151a5d) by running the source directly, and on the published 4.1.1 package.

Reproduction

import { encode, decode } from '@toon-format/toon'

decode(encode('8'))    // 8        (number, expected the string "8")
decode(encode('abc'))  // "abc"    (expected "abc")
decode(encode(''))     // {}       (expected "")

Encoded output for each is the raw string, e.g. encode('8') returns "8" with no quotes.

Only the root primitive position is affected. Keys starting with U+FEFF are already quoted by encodeKey (non-ASCII fails the unquoted-key pattern), and inline array values, tabular cells, list items, and object field values never start the document:

decode(encode({ 'k': 1 }))  // { 'k': 1 }  ok
decode(encode(['x']))       // ['x']       ok
decode(encode({ k: 'x' }))  // { k: 'x' }  ok

Why this is a bug

  • SPEC §12: "A U+FEFF anywhere else is content. Encoders MUST NOT emit one." A document that begins with U+FEFF has, by the spec's own definition, been emitted with a BOM.
  • SPEC §2: decode(encode(x)) must equal x under JSON-model equality.
  • The encoder already knows about this trap: RawString in packages/toon/src/encode/raw-string.ts rejects raw strings that would read as a comment line after a leading BOM is removed. The normal string path in isSafeUnquoted (packages/toon/src/shared/validation.ts) has no equivalent check.

Suggested fix

Two options, either is a one-liner:

  1. In encodeJsonValue (packages/toon/src/encode/encoders.ts), when the root value is a string that starts with U+FEFF, force the quoted form. This keeps the change scoped to the only position where it matters.
  2. Add value.startsWith(BYTE_ORDER_MARK) to isSafeUnquoted. Simpler, but it also quotes such strings in positions where it isn't needed.

Option 1 seems closer to the spirit of "quote only when required" (§7.2). Happy to open a PR for whichever you prefer.

It may also be worth a note in §7.2 of the spec, since the current quoting rules do not mention U+FEFF and any implementation that follows them literally will have the same gap. The Rust port has the identical bug and a fix has been requested there.

Activity

  1. shreyasbhat0 commented on Sep 15, 2026

    @shreyasbhat0
    MemberAuthor

    @johannschopplich flagging this for you since it touches the quoting rules and probably the spec text as well. Happy to send a PR once you've picked an approach.

  2. johannschopplich commented on Oct 1, 2026

    @johannschopplich
    Member

    Went with option 1 – #343 quotes at the root only, and toon-format/spec#60 adds the case to §7.2 with fixtures, so toon-rust#79 can test against those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions