Repository navigation
Description
Description
I'm working on a client library for an API product of some multibillion dollar company, that seems to use some proprietary server software, probably from another multibillion dollar company. Neither of them seem to give a .... have the necessary diligence, to make their products fully JSON specification compatible. Or some other top secret business reasons, like not enough budget for some programmer hours, to fix it.
Anyways, what I want to say, it's something that probably won't change and is something the client side will have to live and deal with. And it's probably something, that shouldn't be too uncommon to be found out there in the "wild".
What you irregularly find in these API results is "null" instead of null, "123" instead of 123, "123.45" instead of 123.45and so on. Of course not consistently, some response fields have the erroneous quotation always, some occasionally, and some are encoded correctly. So far the best approach I found to deal with it, is this:
import msgspec
class IntStr(int):
pass
class FloatStr(float):
pass
def encode_quoted(obj):
if isinstance(obj, int):
return int(obj)
if isinstance(obj, float):
return float(obj)
raise TypeError(f"Unencodable type: {type(obj).__name__}: {obj}")
def decode_quoted(type_, obj):
if type_ is IntStr:
if isinstance(obj, int):
return IntStr(obj)
if obj == "null":
return IntStr(0)
return IntStr(obj.strip("'\" "))
if type_ is FloatStr:
if isinstance(obj, (float, int)):
return FloatStr(obj)
if obj == "null":
return FloatStr("nan")
return FloatStr(obj.strip("'\" "))
raise TypeError(f"Undecodable type: {type(obj).__name__}: {obj}. Expected type: {type_.__name__}")Now, unfortunately this only partly does, what it's supposed to do. I can declare IntStr fields, and they get parsed to an IntStr object. Ideally, I'd like to have them plain int, but this doesn't work. If I try to return int instead of IntStr from the decode hook, I get a TypeError
Also it has issues with optional values:
>>> msgspec.json.decode(b'"123"', type=IntStr, dec_hook=decode_quoted)
123
>>> msgspec.json.decode(b'"null"', type=Optional[IntStr], dec_hook=decode_quoted)
0Ideally, I'd like to have None here, but again, I get a TypeError, if I try to return some other type than IntStr.
So far I also failed to define a Union as a custom type for the decode hook. If there is any possibility, that would already be a big help.
I'm not sure what I should suggest as a best solution for this. Maybe another keyword to msgspec.field and msgspec.Meta like attempt_unquote=? Or the possibility of decode_hook to return other types, than the ones that were specified in the type definition?
Activity
[-]Handling of misformatted JSON[/-][+]Handling of misformatted JSON / hook extension[/+]jcrist commentedon Jun 3, 2023
Luckily all of the cases above match our current coercion rules for the
str_values=Trueoption tomsgspec.from_builtins. I think what I'd want to do here is:from_builtinsoption fromstr_values=Truetocoerce=True(or maybestrict=False? not sure on a good name, thoughts welcome here)In your case then you could continue to use the standard type definitions:
[-]Handling of misformatted JSON / hook extension[/-][+]Handling of misformatted JSON[/+]jcrist commentedon Jun 11, 2023
This has been resolved in #434. When decoding, you can pass
strict=Falseto eitherdecodeor when creating aDecoderto enable a wider set of coercion rules.I expect to cut a release in a week or so, but if you want to try this out before then you can install from github.
ml31415 commentedon Jun 12, 2023
Thanks for the quick fix!
jcrist commentedon Jun 13, 2023
This has been released as part of 0.16.0, available on pypi and conda-forge now.