Skip to content

Clarify msgspec.Raw unsupported with yaml/toml decode - #1170

Closed
Brohammad wants to merge 4 commits into
msgspec:mainfrom
Brohammad:fix-raw-yaml-decode-error
Closed

Brohammad wants to merge 4 commits into
msgspec:mainfrom
Brohammad:fix-raw-yaml-decode-error

Conversation

@Brohammad

Copy link
Copy Markdown

Summary

  • Raise a clear ValidationError when convert targets msgspec.Raw but receives an already-parsed Python object (as happens with yaml.decode / toml.decode)
  • Document that delayed decoding with msgspec.Raw requires the native json or msgpack decoders

Fixes #1136

Root cause

msgspec.Raw requires substrings of the original serialized bytes. The native json and msgpack decoders capture these via *_decode_raw(). The yaml and toml modules parse messages fully with PyYAML/tomllib before calling convert, so the original byte spans are unavailable.

Previously, convert dispatched on the source Python type and produced a confusing error like Expected \any`, got `int``.

Test plan

  • Added test_raw_from_parsed_object_error in test_convert.py
  • Added test_decode_raw_field_unsupported in test_yaml.py
  • Existing Raw tests for json/msgpack/convert still pass (805 tests)

When convert targets msgspec.Raw but the input is an already-parsed
Python object (as with yaml/toml decode), raise a clear ValidationError
instead of a confusing "Expected `any`, got `int`" message. Document
that delayed decoding with Raw requires json or msgpack.

Fixes msgspec#1136

This comment was marked as low quality.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
@Brohammad

Copy link
Copy Markdown
Author

Fork workflow runs need maintainer approval (Build and Test, docs pr, CodSpeed are all action_required). The Deploy preview failure is expected until docs pr is approved and completes — not a code issue. Ready for review when workflows are approved.

@Siyet Siyet left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found no correctness blocker for the field-value case in #1136. Please coordinate this change with #1155 and #1169:

  • In #1155, MS_TYPE_RAW is used for Raw, so the type->types == 0 condition will no longer match this type. Whichever PR is merged second will need an updated condition, preserving Raw/None union behavior.
  • String dictionary keys are handled through convert_str() without a call to convert(). Therefore, the error for convert({"x": 1}, dict[msgspec.Raw, int]) remains Expected any, got str in this version. That diagnostic is corrected in #1169. The two fixes cover different execution paths.

@codspeed

codspeed Bot commented Sep 8, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 139 untouched benchmarks
⏩ 135 skipped benchmarks1


Comparing Brohammad:fix-raw-yaml-decode-error (de14bfb) with main (3f67b34)

Open in CodSpeed

Footnotes

  1. 135 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@Siyet

Siyet commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Thx for the contribution. We merged #1169 to address #1136 with a consistent error message for both values and dictionary keys. Closing this as an alternative implementation

@Siyet Siyet closed this Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

msgspec.Raw appears not to work with msgspec.yaml.decode

3 participants