Repository navigation
fix: correct misleading "Expected any" error for Raw fields in convert() - #1169
Merged
Merged
Conversation
…nvert() msgspec.Raw fields are marked with a typecode of 0. JSON/msgpack decoding never reaches typenode_simple_repr for a Raw mismatch, since the raw bytes are captured before type checking. convert() has no such interception, so a non-Raw input against a Raw-typed field falls through to the generic error path, which reported the expected type as "any" -- directly contradicting the ValidationError being raised. Report the expected type as "raw" instead, matching the wording already used for the reverse case (an already-Raw input rejected by a non-Raw target). Also documents that Raw fields can only be populated by json.decode/msgpack.decode, not by convert() (and thus not by yaml.decode/toml.decode), since there's no encoded-byte buffer left to capture once PyYAML/tomllib have already parsed the input. Fixes msgspec#1136.
Expected any`` error for Raw fields in convert()
3 tasks done
Siyet
approved these changes
Sep 9, 2026
Contributor
|
Thx for the fix! The error message and documentation are much clearer now. |
Siyet
added this pull request to the merge queue
Sep 9, 2026
This was referenced Sep 9, 2026
pull Bot
pushed a commit
to Mu-L/msgspec
that referenced
this pull request
Sep 10, 2026
Invalid input for required `Raw` fields in a `TypedDict`, or `Raw` fields with a dataclass default factory, could produce an empty expected-type name in validation errors. Ignore field metadata when formatting the type name so these errors include `raw`. Follow-up to msgspec#1169. Co-authored-by: Siyet <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #1136.
A
Raw-typed field fed a plain decoded value throughconvert()(and therefore throughyaml.decode/toml.decode, both of which parse first and callconvert()on the result) raises aValidationErrorthat says the expected type isany:That's self-contradictory -- if
anywere really acceptable, there'd be no error at all.Root cause
msgspec.Rawfields are marked with a typecode of0(see the comment at_core.c's type-node builder: "Raw is marked with a typecode of 0, nothing to do"). Forjson.decode/msgpack.decode, that's fine: the raw byte span for aRawfield is captured up front, before any type-checking againsttype->typeshappens, so a typecode-0 mismatch is never actually reached through those paths.convert()has no equivalent interception -- it dispatches purely on the input Python object's type (convert_int,convert_str,convert_dict, ...), and each of those falls through to a generic "no match" error when none of the target'stype->typesbits are set. That generic path callstypenode_simple_repr()to build the human-readable "expected" string, and that function treats typecode0identically to realAny(bitMS_TYPE_ANYset), returning"any"for both. RealAnyfields always set theMS_TYPE_ANYbit explicitly (state->types = MS_TYPE_ANY), sotypes == 0is unambiguously theRawmarker -- but the repr function was conflating the two.The one already-supported case (
convert({"x": raw_instance}, type=Ex), where the input is already aRawobject) goes through a separateconvert_rawdispatch keyed on the input's Python type, and correctly labels a rejected input as`raw`in the reverse direction. This PR makes the forward direction consistent with that existing wording.What changed
src/msgspec/_core.c:typenode_simple_repr()now returns"raw"for typecode0instead of"any", so the error readsExpected `raw`, got `int`-- accurate, and consistent with the wording already used for the reverse mismatch direction. RealAny-typed fields are unaffected (confirmed they always carry the explicitMS_TYPE_ANYbit, never falling into thetypes == 0branch).tests/unit/test_convert.py: added a parametrized regression test (TestRaw.test_raw_mismatch_error_message) covering int/str/array/object inputs against aRaw-typed field.docs/supported-types.rst: added a note under theRawsection clarifying thatRawfields are only populated byjson.decode/msgpack.decode(which can capture original encoded bytes before decoding further);convert()and anything built on it (yaml.decode,toml.decode) operates on already-parsed Python objects, so there's no encoded-byte buffer left to capture, and only an already-constructedRawobject is accepted as input.Explicitly out of scope
The issue (filed as a question) could also be read as "should
convert()support building aRawfrom an arbitrary already-decoded value?" -- e.g. by re-serializing it to some byte format. I deliberately didn't attempt that here:Raw's whole documented purpose is "already-encoded bytes, so the encoder can skip re-encoding it," and a plain Python dict decoded from YAML isn't already encoded in any particular wire format, so picking one (JSON? msgpack?) to synthesize would be inventing new semantics rather than fixing a bug. If that's wanted, it feels like a separate design decision -- happy to hear ifRawis intended to be constructible viaconvert()at all before attempting it.How it was verified
msgspec.Rawappears not to work withmsgspec.yaml.decode#1136 (yaml.decodewith adict[str, msgspec.Raw]field, five different YAML scalar/collection shapes) against unmodifiedmainfirst, confirming the misleadingExpected `any`message.Expected `raw`, got `<type>`as expected; the existing supported case (convert({"x": raw_instance}, type=Ex)) still round-trips correctly; a genuineAny-typed field mismatch is unaffected.just rebuild=1 test-all: 6414 passed, 113 skipped (0 failed). Ran the identical command against unmodifiedmainfor comparison: 6410 passed, 113 skipped -- the +4 delta is exactly the new parametrized regression test (int/str/array/object), zero regressions.just test-doc: 7 passed (doctests unaffected).just hooks-check(ruff lint, ruff format, codespell): all passed.just doc-build: Sphinx build succeeds cleanly with the newsupported-types.rstnote.