Repository navigation
More info in error messages and error handling #420
Description
Description
Error reporting
I have JSON structures, that are rather lengthy and rather nested. If some error occurs somewhere in dozens of MB of JSON data, it's a bit annoying, to dig into those files and understand what exactly caused the error. The msgspec.ValidationError message currently states, which validation failed, which is a great start, but it doesn't show the raw data in question. Also, it's unfortunately not possible to step right into the action with the debugger.
msgspec.ValidationError: Expected `str` of length <= 3 - at `$[1688].blabla[0].metadata.COUNTRY_CODE`
Having the validation violating data printed right in the error message would save a ton of digging deep down in those JSON structs, in order to understand, what's wrong. Trimming down of the raw error value might of course be necessary occasionally. #264 is related.
Error handling
It would also be great, to have the option to set something like errors="ignore" on a per-field and a per-struct level. This would in general allow, to have a stricter validation with customized types, e.g. something like CountryCode instead of a generic str, while relying on a default value, if something goes wrong, instead of plain crashing. "warn" might be another behaviour option.
Activity
jcrist commentedon Jun 3, 2023
Thanks for opening this. The reason the error message doesn't include the invalid data is for security reasons. All tracebacks raised by msgspec only include information written by the developer (e.g. the schema), not information provided by the failing payload. Since tracebacks may be logged or reported in various contexts, I wanted to make sure that any data specific to a message isn't part of the default traceback (where it might leak security relevant data).
That said, I can see how having this extra context available as attributes on the error could be useful. I'll have to think a bit to see what information is possible to include without sacrificing performance. We can definitely do better here.
One tip on debugging these issues: if you load the data without a specific type annotation
msgspec.json.decode(data)the ordering and indices will be identical to those above. So you could find the faulty data as:ml31415 commentedon Jun 3, 2023
Hi Jim! Thanks for your answer. Yeah, that's understandable to have that as a default for security reasons. And yes, that's also kind of what I ended up doing. How about a
verbose_errors=boolas a keyword indecodeinstead?The thing that bugs me more, is the second part, error handling. Because that's something I can not work around otherwise, except having a costly second pass over all data. Would be great, if you could also comment on that.
jcrist commentedon Jun 3, 2023
Part of msgspec's design is coming up with sane defaults, rather than making everything configurable. Every configuration option increases the maintenance burden, and can make things more confusing to users ("what should I pass for this option?"). I don't want to have the sea of config options that pydantic does.
I'd rather attach some extra info that can be included as attributes on the exception by default, but won't be rendered in the exception message. You can then access these extra attributes when debugging, or to handle failures in a custom way.
ml31415 commentedon Jun 3, 2023
Sounds great for me with the extra info on the error object. It would also allow to customize yourself, if desired.
Ok, what about the second part of the question, the error handling? Let me detail it a bit more: I got thousands of entries, that pass validation, e.g. country code as mentioned above. And then there is this one outlier, a wrong entry, that the other side shouldn't have put in there. I'd rather have that value to be set to
None, the default value, instead of having a decoding error thrown. What about such an option? Or what would you suggest, to parse "everything else, except that lonely wrong field"?jcrist commentedon Jun 3, 2023
We could support this option, but that'd be a separate issue than the above. I'll open a new issue.
jcrist commentedon Jun 3, 2023
See #425
ml31415 commentedon Jun 3, 2023
Yeah, I probably should have split it up from the beginning. Though, I had felt a bit bad for "issue flooding" already, as
msgspecis really a diamond of a library. Big thanks for the great work!ml31415 commentedon Jan 27, 2024
If the raw data is never supposed to be presented in error messages, this is in fact a duplicate of #264 .