Skip to content

Handling of misformatted JSON #424

Closed

Description

@ml31415

Description

I'm working on a client library for an API product of some multibillion dollar company, that seems to use some proprietary server software, probably from another multibillion dollar company. Neither of them seem to give a .... have the necessary diligence, to make their products fully JSON specification compatible. Or some other top secret business reasons, like not enough budget for some programmer hours, to fix it.

Anyways, what I want to say, it's something that probably won't change and is something the client side will have to live and deal with. And it's probably something, that shouldn't be too uncommon to be found out there in the "wild".

What you irregularly find in these API results is "null" instead of null, "123" instead of 123, "123.45" instead of 123.45and so on. Of course not consistently, some response fields have the erroneous quotation always, some occasionally, and some are encoded correctly. So far the best approach I found to deal with it, is this:

import msgspec

class IntStr(int):
    pass
    
class FloatStr(float):
    pass


def encode_quoted(obj):
    if isinstance(obj, int):
        return int(obj)
    if isinstance(obj, float):
        return float(obj)
    raise TypeError(f"Unencodable type: {type(obj).__name__}: {obj}")


def decode_quoted(type_, obj):
    if type_ is IntStr:
        if isinstance(obj, int):
            return IntStr(obj)
        if obj == "null":
            return IntStr(0)
        return IntStr(obj.strip("'\" "))
    if type_ is FloatStr:
        if isinstance(obj, (float, int)):
            return FloatStr(obj)
        if obj == "null":
            return FloatStr("nan")
        return FloatStr(obj.strip("'\" "))
    raise TypeError(f"Undecodable type: {type(obj).__name__}: {obj}. Expected type: {type_.__name__}")

Now, unfortunately this only partly does, what it's supposed to do. I can declare IntStr fields, and they get parsed to an IntStr object. Ideally, I'd like to have them plain int, but this doesn't work. If I try to return int instead of IntStr from the decode hook, I get a TypeError

Also it has issues with optional values:

>>> msgspec.json.decode(b'"123"', type=IntStr, dec_hook=decode_quoted)
123
>>> msgspec.json.decode(b'"null"', type=Optional[IntStr], dec_hook=decode_quoted)
0

Ideally, I'd like to have None here, but again, I get a TypeError, if I try to return some other type than IntStr.

So far I also failed to define a Union as a custom type for the decode hook. If there is any possibility, that would already be a big help.

I'm not sure what I should suggest as a best solution for this. Maybe another keyword to msgspec.field and msgspec.Meta like attempt_unquote=? Or the possibility of decode_hook to return other types, than the ones that were specified in the type definition?

Activity

  1. changed the title [-]Handling of misformatted JSON[/-] [+]Handling of misformatted JSON / hook extension[/+] on Jun 2, 2023
  2. jcrist commented on Jun 3, 2023

    @jcrist
    Member

    Luckily all of the cases above match our current coercion rules for the str_values=True option to msgspec.from_builtins. I think what I'd want to do here is:

    • Rename the from_builtins option from str_values=True to coerce=True (or maybe strict=False? not sure on a good name, thoughts welcome here)
    • Add this option to all builtin decoders. When enabled, we'd support more coercion of types to try and fit the provided schema.

    In your case then you could continue to use the standard type definitions:

    msgspec.json.decode(msg, type=YourType, coerce=True)
  3. changed the title [-]Handling of misformatted JSON / hook extension[/-] [+]Handling of misformatted JSON[/+] on Jun 4, 2023
  4. jcrist commented on Jun 11, 2023

    @jcrist
    Member

    This has been resolved in #434. When decoding, you can pass strict=False to either decode or when creating a Decoder to enable a wider set of coercion rules.

    I expect to cut a release in a week or so, but if you want to try this out before then you can install from github.

  5. ml31415 commented on Jun 12, 2023

    @ml31415
    Author

    Thanks for the quick fix!

  6. jcrist commented on Jun 13, 2023

    @jcrist
    Member

    This has been released as part of 0.16.0, available on pypi and conda-forge now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions