Skip to content

fix: correct misleading "Expected any" error for Raw fields in convert() - #1169

Merged
Siyet merged 1 commit into
msgspec:mainfrom
Ananthr16:fix/raw-convert-error-message
Sep 9, 2026
Merged

Siyet merged 1 commit into
msgspec:mainfrom
Ananthr16:fix/raw-convert-error-message

Conversation

@Ananthr16

Copy link
Copy Markdown
Contributor

Problem

Fixes #1136.

A Raw-typed field fed a plain decoded value through convert() (and therefore through yaml.decode/toml.decode, both of which parse first and call convert() on the result) raises a ValidationError that says the expected type is any:

>>> class Repro(msgspec.Struct):
...     tools: dict[str, msgspec.Raw]
>>> msgspec.yaml.decode(b"tools:\n  x: 1\n", type=Repro)
msgspec.ValidationError: Expected `any`, got `int` - at `$.tools[...]`

That's self-contradictory -- if any were really acceptable, there'd be no error at all.

Root cause

msgspec.Raw fields are marked with a typecode of 0 (see the comment at _core.c's type-node builder: "Raw is marked with a typecode of 0, nothing to do"). For json.decode/msgpack.decode, that's fine: the raw byte span for a Raw field is captured up front, before any type-checking against type->types happens, so a typecode-0 mismatch is never actually reached through those paths.

convert() has no equivalent interception -- it dispatches purely on the input Python object's type (convert_int, convert_str, convert_dict, ...), and each of those falls through to a generic "no match" error when none of the target's type->types bits are set. That generic path calls typenode_simple_repr() to build the human-readable "expected" string, and that function treats typecode 0 identically to real Any (bit MS_TYPE_ANY set), returning "any" for both. Real Any fields always set the MS_TYPE_ANY bit explicitly (state->types = MS_TYPE_ANY), so types == 0 is unambiguously the Raw marker -- but the repr function was conflating the two.

The one already-supported case (convert({"x": raw_instance}, type=Ex), where the input is already a Raw object) goes through a separate convert_raw dispatch keyed on the input's Python type, and correctly labels a rejected input as `raw` in the reverse direction. This PR makes the forward direction consistent with that existing wording.

What changed

  • src/msgspec/_core.c: typenode_simple_repr() now returns "raw" for typecode 0 instead of "any", so the error reads Expected `raw`, got `int` -- accurate, and consistent with the wording already used for the reverse mismatch direction. Real Any-typed fields are unaffected (confirmed they always carry the explicit MS_TYPE_ANY bit, never falling into the types == 0 branch).
  • tests/unit/test_convert.py: added a parametrized regression test (TestRaw.test_raw_mismatch_error_message) covering int/str/array/object inputs against a Raw-typed field.
  • docs/supported-types.rst: added a note under the Raw section clarifying that Raw fields are only populated by json.decode/msgpack.decode (which can capture original encoded bytes before decoding further); convert() and anything built on it (yaml.decode, toml.decode) operates on already-parsed Python objects, so there's no encoded-byte buffer left to capture, and only an already-constructed Raw object is accepted as input.

Explicitly out of scope

The issue (filed as a question) could also be read as "should convert() support building a Raw from an arbitrary already-decoded value?" -- e.g. by re-serializing it to some byte format. I deliberately didn't attempt that here: Raw's whole documented purpose is "already-encoded bytes, so the encoder can skip re-encoding it," and a plain Python dict decoded from YAML isn't already encoded in any particular wire format, so picking one (JSON? msgpack?) to synthesize would be inventing new semantics rather than fixing a bug. If that's wanted, it feels like a separate design decision -- happy to hear if Raw is intended to be constructible via convert() at all before attempting it.

How it was verified

  • Reproduced the exact scenario from msgspec.Raw appears not to work with msgspec.yaml.decode #1136 (yaml.decode with a dict[str, msgspec.Raw] field, five different YAML scalar/collection shapes) against unmodified main first, confirming the misleading Expected `any` message.
  • After the fix, all five now raise Expected `raw`, got `<type>` as expected; the existing supported case (convert({"x": raw_instance}, type=Ex)) still round-trips correctly; a genuine Any-typed field mismatch is unaffected.
  • just rebuild=1 test-all: 6414 passed, 113 skipped (0 failed). Ran the identical command against unmodified main for comparison: 6410 passed, 113 skipped -- the +4 delta is exactly the new parametrized regression test (int/str/array/object), zero regressions.
  • just test-doc: 7 passed (doctests unaffected).
  • just hooks-check (ruff lint, ruff format, codespell): all passed.
  • just doc-build: Sphinx build succeeds cleanly with the new supported-types.rst note.

…nvert()

msgspec.Raw fields are marked with a typecode of 0. JSON/msgpack decoding
never reaches typenode_simple_repr for a Raw mismatch, since the raw bytes
are captured before type checking. convert() has no such interception, so
a non-Raw input against a Raw-typed field falls through to the generic
error path, which reported the expected type as "any" -- directly
contradicting the ValidationError being raised.

Report the expected type as "raw" instead, matching the wording already
used for the reverse case (an already-Raw input rejected by a non-Raw
target). Also documents that Raw fields can only be populated by
json.decode/msgpack.decode, not by convert() (and thus not by
yaml.decode/toml.decode), since there's no encoded-byte buffer left to
capture once PyYAML/tomllib have already parsed the input.

Fixes msgspec#1136.
@Ananthr16 Ananthr16 changed the title fix: correct misleading Expected any`` error for Raw fields in convert() fix: correct misleading "Expected any" error for Raw fields in convert() Sep 2, 2026
@Siyet

Siyet commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Thx for the fix! The error message and documentation are much clearer now.

@Siyet
Siyet added this pull request to the merge queue Sep 9, 2026
Merged via the queue into msgspec:main with commit 8be1966 Sep 9, 2026
27 of 28 checks passed
pull Bot pushed a commit to Mu-L/msgspec that referenced this pull request Sep 10, 2026
Invalid input for required `Raw` fields in a `TypedDict`, or `Raw`
fields with a dataclass default factory, could produce an empty
expected-type name in validation errors. Ignore field metadata when
formatting the type name so these errors include `raw`.

Follow-up to msgspec#1169.

Co-authored-by: Siyet <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

msgspec.Raw appears not to work with msgspec.yaml.decode

2 participants