Support using default value if provided data was invalid, rather than erroring #425

Activity

1. jcrist commented on Jun 3, 2023

   

   jcrist

   on Jun 3, 2023

   MemberAuthor

   More actions

   I'm not sure if this is an option I'd want to support, I think it would add a lot of complexity to the codebase. I'm not sure how much utility there is in ignoring invalid data rather than loosening your validation and handling it downstream of the decoder.

   If we did support it, it'd make sense to do this by adding another option to msgspec.Meta annotations. This way it could be applied to any type node, not just a top-level field. Something like:

   class Message(msgspec.Struct):
       country_codes: list[Annotated[CountryCode, Meta(on_error=CountryCode.INVALID)]]
   
   msgspec.json.decode('{"country_codes": ["USA", "CAN"]}')
   #> Message(country_codes=[CountryCode.USA, CountryCode.CAN])
   
   msgspec.json.decode('{"country_codes": ["OOOPS"]}')
   #> Message(country_codes=[CountryCode.INVALID])
2. jcrist commented on Jun 3, 2023

   

   jcrist

   on Jun 3, 2023

   MemberAuthor

   More actions

   In this example case above we're handling an invalid Enum values. A simpler case for only enums would be to make use of the stdlibs _missing_ enum hook to provide a default value. We could add support for this for enum types, which would be a lot easier than adding a generic on_error option.

   The above could then be written:

   from enum import Enum
   
   class CountryCode(Enum):
       USA = "USA"
       CAN = "CAN"
       INVALID = "INVALID"
       # ....
       @classmethod
       def _missing_(cls, value):
           return cls.INVALID
   
   class Message(msgspec.Struct):
       country_codes: list[CountryCode]
   
   msgspec.json.decode('{"country_codes": ["USA", "CAN"]}')
   #> Message(country_codes=[CountryCode.USA, CountryCode.CAN])
   
   msgspec.json.decode('{"country_codes": ["OOOPS"]}')
   #> Message(country_codes=[CountryCode.INVALID])
3. ml31415 commented on Jun 3, 2023

   

   ml31415

   on Jun 3, 2023

   Last edited by ml31415

   More actions

   The on_error= keyword sounds good for me. But the solution shouldn't be limited to enums. It would be great, if on_error= could additionally to an error value, also be a simple function, that would take the invalid, but parsed object, and would have to create some result instead, or optionally still throw a ValueError / Validation error.
4. ml31415 commented on Jun 4, 2023

   

   ml31415

   on Jun 4, 2023

   More actions

   This might be combined with the coerce= keyword from #424 (comment) : coerce could not only be a bool value, but such a fix-up-function, that could be specifically provided, if the inbuilt coercion mechanisms fail. So, coerce=func instead of on_error=....
5. jcrist commented on Jun 12, 2023

   

   jcrist

   on Jun 12, 2023

   MemberAuthor

   More actions

   I'm not sure if this is an option I'd want to support, I think it would add a lot of complexity to the codebase. I'm not sure how much utility there is in ignoring invalid data rather than loosening your validation and handling it downstream of the decoder.

   Closing this, I no longer think this is a good idea. The same mechanism can be achieved by loosening the validation rules for a field on initial decoding, and handling validation issues downstream. The downstream usage might make use of msgspec.convert to do further validation, and returning a default value on ValidationError.
6. jcrist

   closed this as completedon Jun 12, 2023
7. ml31415 commented on Jun 13, 2023

   

   ml31415

   on Jun 13, 2023

   More actions

   That's a pity to hear. As you say, it means falling back to "loosening" i.e. scratching validation besides a type check. I understand that it increases complexity, but so does having validation in general.

   Maybe the above example with the country codes was not convincing for you, but I really think it's an important feature in general, that shouldn't be left aside easily.

   E.g. have a look the python inbuilt codes package. It's quite the same situation there. You might have a ton of well behaved text / data, and then the one lonely outlier, that might spoil a ton of otherwise valid data. What to do? Ditch the ton of data because of the erroneous outlier? Or accept it and go on. What are the options, if you need the other data?

   Write a separate parser, that tries to scratch out all other readable parts on your own? Ok, but why reinvent the wheel? And why have two libraries, that do the same job for 99% of the time? Yes that separate parser might make use of msgspec, but it would have to slowly iterate through lists and dicts, try to convert them step by step etc. it would be slow and error prone.

   It should be the users choice, what kind of error handling he wants to have. Safe and strict defaults are totally fine, but it needs to be adjustable. There is no point in insisting on correctness, if your data source isn't, and you can't fix it.

   codecs defines strict, ignore and different versions of replace as error handling behaviour for decode(obj, errors=...). It would be great, if the same would be possible for msgspec.